ADF PostgreSQL Metadata Lookup fails with Npgsql connection pool exhausted despite only ~20 PostgreSQL connections

ADF PostgreSQL Metadata Lookup fails with Npgsql connection pool exhausted despite low PostgreSQL connections

Post

I’m troubleshooting an Azure Data Factory (ADF) pipeline where a PostgreSQL Metadata Lookup activity is failing with an Npgsql connection pool exhaustion error:

NpgsqlException: The connection pool has been exhausted, either raise MaxPoolSize (currently 800) or Timeout (currently 600 seconds).

The PostgreSQL linked service connects to an Azure Database for PostgreSQL Flexible Server.

Current configuration:

• PostgreSQL max_connections: 3437
• PgBouncer: Disabled
• ADF PostgreSQL linked service:
  • Maximum Pool Size = 800
    • Timeout = 600 seconds

The Maximum Pool Size and Timeout values were manually configured in the ADF linked service.

Previously, the Metadata Lookup completed in less than one minute. Recently, it takes approximately 5–7 minutes and eventually fails with the connection pool exhaustion error.

However, PostgreSQL itself does not appear to be under connection pressure. The observed connection counts are approximately 8, 6, 5, and 1, totaling around 20 connections. The Azure PostgreSQL Active Connections metric also appears normal and is nowhere near the configured max_connections limit.

The PostgreSQL server is shared by multiple applications. I can see connections from different users/applications, but I don't have sufficient privileges to inspect the detailed sessions for some of them.

Questions

1. Is Maximum Pool Size = 800 in the ADF PostgreSQL linked service an Npgsql client-side connection pool, separate from PostgreSQL's max_connections?
2. How can I determine how many connections are actually being created or held by the ADF PostgreSQL linked service?
3. How can I determine whether connections are being leaked or not returned to the Npgsql pool?
4. Are there any ADF or Azure PostgreSQL diagnostics/logs that can help trace the connection flow from ADF → Npgsql → PostgreSQL?
5. Could the 5–7 minute Metadata Lookup execution time indicate connections waiting or being held somewhere?
6. Is there any known ADF PostgreSQL connector/Npgsql pooling behavior that could cause the client-side pool to become exhausted while PostgreSQL shows only a small number of connections?

I’m looking for guidance on identifying the root cause, rather than simply increasing the MaxPoolSize further.