Microsoft hosted Agent Ubuntu VMs – Python Dependency Installation Failure

Gokul Sandar Muthuramalingam 0 Reputation points
2026-11-20T12:16:33.62+00:00

Our Azure DevOps pipeline running on Microsoft-hosted Ubuntu agents is unable to establish outbound HTTPS connections to the public Python package repositories.

The following endpoints consistently time out:

https://files.pythonhosted.org/

Our diagnostics confirm:

DNS resolution is successful.

General HTTPS connectivity works (Google returns HTTP 200).

Both IPv4 requests time out; IPv6 connections fail.

No HTTP/HTTPS proxy is configured.

The issue occurs on both host-level and Docker-based requests.

The issue persists across Ubuntu 22.04 and 24.04 and both BuildKit and legacy Docker builders.

Please investigate outbound network routing/connectivity from Microsoft-hosted Ubuntu agents to PyPI and Python package hosting infrastructure.

Azure DevOps
0 comments No comments

1 answer

Sort by: Most helpful
  1. AI answer

    2026-11-20T12:17:41.4333333+00:00

    No broad Azure DevOps or PyPI incident is currently reported, so capture the hosted agent’s per-job egress and TCP/TLS evidence and open an Azure DevOps support request for an endpoint-specific outbound routing investigation.

    Microsoft-hosted jobs receive a fresh VM for every job, and Ubuntu agents can be placed in any Azure region mapped to the Azure DevOps organization’s geography.^1^ Consequently, results must include the failing job’s apparent egress address, timestamps, and resolved PyPI addresses so Microsoft can correlate the affected route.

    The current public signals show Azure DevOps Pipelines as healthy and Python reports pypi.org, its CDN, and files.pythonhosted.org as operational.^2^^3^ These dashboards do not exclude a localized route, region, or egress-path failure.

    Add a single-job evidence capture

    Since DNS, IPv4/IPv6, Docker, both builders, and Ubuntu 22.04/24.04 have already been tested, avoid repeating those permutations. Add this step immediately before dependency installation:

    variables:
      system.debug: 'true'
    
    steps:
    - bash: |
        set +e
        echo "UTC=$(date -u --iso-8601=seconds)"
        echo "Agent=$(Agent.Name) OS=$(Agent.OS)"
        echo "Public egress:"
        curl -4 -sS --connect-timeout 10 https://api.ipify.org; echo
        for host in pypi.org files.pythonhosted.org; do
          echo "===== $host ====="
          getent ahosts "$host"
          curl -4 -vI --connect-timeout 15 --max-time 30 "https://$host/" 2>&1
          tracepath -n "$host" 2>&1 || true
        done
      displayName: Capture PyPI network evidence
      condition: always()
    

    If use of api.ipify.org is not permitted by policy, omit that request; the remaining output is still useful.

    System.Debug: true enables verbose pipeline logging.^4^ For each failed run, retain:

    • Azure DevOps organization name and organization geography.
    • Pipeline and run IDs, job name, image label, and exact UTC interval.
    • Public egress address, if captured.
    • DNS results, including every returned IPv4/IPv6 address.
    • curl -4 -vI output showing whether failure occurs before TCP connection, during TLS negotiation, or after the HTTP request.
    • tracepath output. Intermediate nonresponses alone do not prove packet loss; the terminal TCP/TLS timeout is the important evidence.
    • Whether a newly queued job receives a different egress path and succeeds.

    Download the complete run logs from Pipeline run → … → Download logs.^5^

    Escalate for route correlation

    Open a support request and classify it under Azure DevOps / Azure Pipelines / Microsoft-hosted agents, describing it as:

    Repeatable outbound TCP 443 timeout from Microsoft-hosted Ubuntu agents to pypi.org and files.pythonhosted.org; DNS succeeds and unrelated HTTPS destinations work. Reproduces on host and in Docker across Ubuntu 22.04/24.04.

    Attach at least two failing runs with exact UTC timestamps and, ideally, one successful control run from another network or self-hosted agent. Azure DevOps documentation directs blocked production issues to Microsoft Support.^6^

    This requires platform-side correlation because Microsoft-hosted agents use Microsoft-controlled public networking, are discarded after each job, and cannot be attached to your VPN or ExpressRoute.^1^

    Temporary mitigation

    If package restoration is blocking delivery, run that job on a self-hosted agent, VM Scale Set agent, or Managed DevOps Pool where outbound routing and NAT are under your control; these are the documented alternatives when Microsoft-hosted agents do not meet networking requirements.^1^

    A private package mirror can also reduce dependency on the live PyPI path, but changing pip timeouts, retries, Docker builders, or Ubuntu images is unlikely to resolve a consistently failing TCP connection.


    References

    1. Microsoft-hosted agents
    2. https://status.dev.azure.com/_apis/status/health?services=Pipelines
    3. Python Infrastructure Status
    4. Use predefined variables
    5. Review logs to diagnose pipeline issues
    6. Get support and provide feedback
    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.