Linux VM stuck in Failed provisioning state (VMStartTimedOut) after data disk hot-attach, guest OS running normally

Harsh, thanks for the details.

The fact that the attach ran for 50 minutes before timing out, and Linux never detected the disk even after a guest reboot, makes this look much more like an Azure-side issue than anything inside the OS.

The disk is recorded in the VM configuration, but it doesn't seem to have been presented to the guest. The later failures with Run Command, reapply, start and detach also suggest that Azure isn't processing operations against this VM correctly.

Since it's still serving production traffic, I'd leave it running and avoid further VM updates for now. You've already tried the usual recovery options, and repeating them while other operations are stuck is unlikely to help.

At this point I'd open an Azure Compute support case and include the correlation IDs and timestamps for the original disk attach and all later failed operations. Ask them to check for stuck backend operations and whether the VM's resource configuration is out of sync with its runtime state.

I'd specifically request a non-disruptive recovery, if possible, and make it clear that deallocation or redeployment isn't an option without an approved maintenance window.

I also wouldn't attempt any further disk detach operations until Microsoft has investigated the current state.

One thing I'd emphasize in the ticket is that the VM remains operational despite repeated VMStartTimedOut errors. That's an important distinction, and I'd ask the Compute team to investigate it before recommending another restart or redeploy.

Collecting diagnostics for the support case

You can use the following read-only PowerShell script from Azure Cloud Shell to collect the VM model, instance view, disk attachment details and Activity Log events.

It doesn't modify any Azure resources. If a diagnostic read fails, the script records the error and continues with the remaining checks.

You can check it on another VM, I have tested on my environment, most of script created by github copilot.

Open Azure Cloud Shell in PowerShell mode, replace the placeholders and run the script. You'll need Reader permissions covering the VM and disk resources, including access to their Activity Logs.

# Azure VM / Disk Diagnostics - READ ONLY
# Requires Az.Accounts, Az.Compute and Az.Monitor

$subscriptionId    = "<subscription-id>"
$resourceGroup     = "<resource-group>"
$diskResourceGroup = $resourceGroup
$vmName            = "<vm-name>"
$diskName          = "<data-disk-name>"

$ErrorActionPreference = "Stop"
$outputFile = Join-Path $HOME "vm-disk-diagnostics.txt"

if (-not (Get-AzContext)) {
    Connect-AzAccount | Out-Null
}

Set-AzContext -SubscriptionId $subscriptionId | Out-Null

$report = [System.Collections.Generic.List[string]]::new()

function Add-Section {
    param(
        [string]$Title,
        [object]$Data
    )

    $script:report.Add("=== $Title ===")

    if ($null -eq $Data) {
        $script:report.Add("No data returned.")
    }
    else {
        $script:report.Add(
            ($Data | Format-List * |
             Out-String -Width 250).TrimEnd()
        )
    }

    $script:report.Add("")
}

# Works with both older (nested) and newer (flat) Az.Monitor output
function Get-Val {
    param($Value)
    if ($null -eq $Value) { return $null }
    if ($Value.PSObject.Properties['LocalizedValue']) { return $Value.LocalizedValue }
    if ($Value.PSObject.Properties['Value'])          { return $Value.Value }
    return [string]$Value
}

# VM model
$vm = Get-AzVM -ResourceGroupName $resourceGroup -Name $vmName

Add-Section "VM MODEL" (
    $vm | Select-Object Name, Location, VmId,
        ProvisioningState,
        @{N="VMSize";E={$_.HardwareProfile.VmSize}}
)

# VM instance view
try {
    $vmStatus = Get-AzVM `
        -ResourceGroupName $resourceGroup `
        -Name $vmName -Status

    Add-Section "VM INSTANCE VIEW" (
        $vmStatus.Statuses |
        Select-Object Code, DisplayStatus, Message, Time
    )
}
catch {
    Add-Section "INSTANCE VIEW ERROR" $_.Exception.Message
}

# Data disks in the VM model
$attachments = @(
    $vm.StorageProfile.DataDisks | ForEach-Object {
        [pscustomobject]@{
            LUN           = $_.Lun
            Name          = $_.Name
            ManagedDiskId = $_.ManagedDisk.Id
            CreateOption  = $_.CreateOption
            Caching       = $_.Caching
        }
    }
)

Add-Section "VM DATA DISKS" $attachments

# Managed disk resource
$disk = $null

try {
    $disk = Get-AzDisk `
        -ResourceGroupName $diskResourceGroup `
        -DiskName $diskName

    Add-Section "DISK RESOURCE" (
        $disk | Select-Object Name, DiskState,
            ManagedBy, Location, ProvisioningState, Id
    )
}
catch {
    Add-Section "DISK READ ERROR" $_.Exception.Message
}

# Activity Log collection
$startTime = (Get-Date).AddDays(-7)

function Add-ActivityLog {
    param(
        [string]$Title,
        [string]$ResourceId
    )

    try {
        $events = Get-AzActivityLog `
            -ResourceId $ResourceId `
            -StartTime $startTime `
            -DetailedOutput `
            -MaxRecord 200

        $items = @(
            $events | Sort-Object EventTimestamp -Descending |
            Select-Object `
                @{N="TimestampUTC";E={([datetime]$_.EventTimestamp).ToUniversalTime().ToString("o")}},
                @{N="Operation";E={Get-Val $_.OperationName}},
                @{N="Status";E={Get-Val $_.Status}},
                @{N="SubStatus";E={Get-Val $_.SubStatus}},
                CorrelationId, ResourceId,
                @{N="Details";E={$_.Properties}}
        )

        Add-Section $Title $items
    }
    catch {
        Add-Section "$Title - ERROR" $_.Exception.Message
    }
}

Add-ActivityLog -Title "VM ACTIVITY LOG" -ResourceId $vm.Id

if ($disk) {
    Add-ActivityLog -Title "DISK ACTIVITY LOG" -ResourceId $disk.Id
}

$report | Out-File -FilePath $outputFile -Encoding utf8

Write-Host "Diagnostics saved to: $outputFile" -ForegroundColor Green
Write-Host "No Azure resources were modified."

If the disk is in a different resource group, update $diskResourceGroup accordingly.

The script saves vm-disk-diagnostics.txt in your Cloud Shell home directory. You can download it using the Cloud Shell Manage files > Download option. If you're using an ephemeral Cloud Shell session, download the file before closing the session.

I'd attach that file to the support case rather than post its complete contents publicly, since it may contain resource identifiers and operational metadata.

The report should give the Azure Compute team a useful starting point, particularly for correlating the failed operations with the VM's reported state. They may still need to investigate the underlying platform telemetry.

At this stage, I'd prioritize getting Microsoft to investigate the VM without disrupting the production workload.

Thanks, that's the key detail. An attach that's accepted, sits for ~50 minutes, never shows up in the guest (no kernel or waagent events, not even after a reboot), and then blocks every later operation points to a stuck operation on the host side, not to anything in Linux. I wouldn't expect it to clear by itself, and further reapply, detach or start attempts will just queue behind it.

For the support case (Azure Compute, Severity A if production is at risk), this collects the timeline they need without changing anything:

az monitor activity-log list --resource-id $(az vm show -g <rg> -n <vm> --query id -o tsv) --offset 3d \
  --query "[].{time:eventTimestamp, op:operationName.localizedValue, status:status.value, corr:correlationId}" -o table

In the case, ask them to:

• clear or complete the stuck attach operation for LUN 1 on the host, and
• confirm a non-disruptive way to bring the VM model back in sync, explicitly stating that deallocate/redeploy needs a maintenance window.

Attach the output to the case rather than posting it here, since it contains resource IDs.

Hi @Vasileios Dionysopoulos
Thanks, I have gone through your suggestions.

Activity Log: the original disk attach (VM update) was accepted at 09:32 UTC and failed with VMStartTimedOut at 10:22 UTC, so it sat for about 50 minutes. I have the correlation IDs and timestamps for it and for the later failed operations (Run Command, two reapplies, a start and a disk detach), and can share them privately with a support engineer.

Guest logs: nothing in the kernel journal or the Azure agent log about a new device around the attach time. The VM has since done its scheduled reboot and still only sees the original data disk, so the attach never reached the guest.

Follow-up results: the second reapply and the disk detach I mentioned both failed with the same VMStartTimedOut after about 50 minutes. Nothing has been run on the VM since. It is still serving production traffic, and the provisioning state is still Failed.

We are on the Developer plan, so I am waiting for a Microsoft support engineer to review this thread. Could someone from the Azure Compute team check for a stuck operation on the host and tell us if the VM can be brought back in sync without a restart or deallocation? A deallocate or redeploy would need an approved maintenance window.