ASA parallel JOIN drops matches when the two Event Hubs inputs use different partition-key hashing

Question

ASA parallel JOIN drops matches when the two Event Hubs inputs use different partition-key hashing

Andrei Josephsen 0

ASA job (compat 1.2) joins two Event Hubs on EntityId (both 4 partitions) into a third hub. With the input Partition key column declared (embarrassingly-parallel), the JOIN silently loses ~78% of matches — no errors. Measured: parallel ≈ 22% match, non-parallel ≈ 100%.

Cause: the two inputs are produced by clients that hash the key differently, so the same EntityId lands on different partitions: EventHubBufferedProducerClient uses client-side lookup3, EventHubProducerClient/gateway uses another hash, and ASA output uses a third.

Q: Can I force all publishers (buffered producer, regular producer, and ASA output) to use the same partition-key hashing function, so the same EntityId always lands on the same partition and a parallel JOIN matches?

SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-25T12:55:26.5066667+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

To investigate this issue further, I have sent you a private message. Could you please provide the following details through the private message?

Error or failure screenshots

Your email address

Time zone

Your availability for a Microsoft Teams meeting

These details will help us perform a deeper investigation and engage the appropriate engineering team if required.

Once you've shared the requested information, we will continue our analysis and provide further guidance.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-26T08:47:01.2333333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
Andrei Josephsen 0 Reputation points

2026-08-26T11:00:46.51+00:00

The reason is fairly clear ... events with a partitionID get routed to different partitions depending on how they were published .. example with four partitions

event with PartitionKey: 8898d388-f719-461f-b26d-2035fbb732f8ends on partition 1 using ASA

ends on partition 2 using sdk EventHub BufferedProducerClient

This basically ruins out combining ASA events with BufferedProducerClient events for high throughput scenarios
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-31T09:55:13.7033333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-02T06:45:13.28+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

While reviewing your support request, we were unable to locate an active entitlement/subscription associated with the account, which is preventing us from creating the DFM case.

Could you please confirm whether you have an active subscription linked to the affected resource? If available, kindly share the Subscription ID or any relevant subscription details so that we can proceed further with the case creation.

We look forward to your response.

Thanks & Regards,
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-02T07:13:24.33+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

Based on the current Azure Event Hubs and Azure Stream Analytics design, it is not possible to force EventHubProducerClient, EventHubBufferedProducerClient, and Azure Stream Analytics outputs to use the same partition-key hashing function.

The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions.

For Azure Stream Analytics parallel JOINs, the engine assumes that records with the same partition key are co-located in corresponding partitions across all inputs. If the upstream producers use different partitioning behaviors and the same EntityId ends up in different partitions, matching events may never be processed together, causing the reduced match rate you observed.

As a workaround, consider one of the following approaches:

• Disable the partition-key optimization and run the JOIN without declaring the Partition key column. This typically restores correctness but may reduce throughput.

• Ensure both input streams are produced through a single consistent publishing path so that identical EntityId values are partitioned consistently before reaching Stream Analytics.

• Repartition or normalize the streams upstream (for example, through an intermediary processing layer) before they are consumed by the ASA job.

Therefore, there is currently no supported mechanism to configure a common partition-key hashing function across EventHubBufferedProducerClient, EventHubProducerClient, and Azure Stream Analytics output destinations.

If maintaining an embarrassingly parallel JOIN is a requirement, the recommended approach is to ensure partition alignment upstream rather than relying on configurable hashing behavior.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-09T08:31:04.5566667+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-11T03:38:48.38+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Andrei Josephsen 0 Reputation points

2026-09-11T08:13:22.95+00:00

"The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions." -

I wish there was some way for users to pick the algorithm - and the preferred would probably be to use the same Murmur2 hashing as Kafka .

Republishing every event, as answer suggest works, but as it ads latency and cost, I would prefer the hashing algorithm to be consistent.

But - answer would be accepted - but as it is just a comment - I don't think I have any way to accept it (technically)
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-12T09:22:28.8866667+00:00
Hi @Andrei Josephsen ,

Since the partition key hash algorithm is an internal implementation detail and cannot currently be configured, there is no supported method to enforce Kafka Murmur2 partition mapping within the service.

The following mitigation options may be considered:

Continue using the existing partition key behavior if cross-platform partition consistency is not a strict requirement.

Republish events through an intermediate processing layer to ensure consistent partition assignment when interoperability with Kafka producers is required.

Standardize message production through a single producer path to minimize differences in partition routing behavior across services.

Submit a product feedback request to the engineering team requesting support for a configurable partition hashing algorithm or Kafka-compatible Murmur2 hashing.

Review application design to determine whether strict partition alignment is required or whether event ordering can be maintained through alternative mechanisms.

Limitation

At present, the partition key hash algorithm is not exposed as a customer-configurable setting. Therefore, different services may map the same partition key value to different physical partitions, and this behavior is expected.

Recommended Resolution

If deterministic partition mapping across Kafka and Azure services is a business requirement, republishing or implementing a custom routing layer remains the most practical workaround. However, this approach introduces additional latency and operational cost. A product enhancement request for configurable hashing algorithms would be the preferred long-term solution.

Thanks & Regards,
Sivasankar.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-16T06:13:45.46+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-17T04:43:51.5933333+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.

Your answer

SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-25T12:55:26.5066667+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

To investigate this issue further, I have sent you a private message. Could you please provide the following details through the private message?

Error or failure screenshots

Your email address

Time zone

Your availability for a Microsoft Teams meeting

These details will help us perform a deeper investigation and engage the appropriate engineering team if required.

Once you've shared the requested information, we will continue our analysis and provide further guidance.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-26T08:47:01.2333333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
Andrei Josephsen 0 Reputation points

2026-08-26T11:00:46.51+00:00

The reason is fairly clear ... events with a partitionID get routed to different partitions depending on how they were published .. example with four partitions

event with PartitionKey: 8898d388-f719-461f-b26d-2035fbb732f8ends on partition 1 using ASA

ends on partition 2 using sdk EventHub BufferedProducerClient

This basically ruins out combining ASA events with BufferedProducerClient events for high throughput scenarios
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-08-31T09:55:13.7033333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-02T06:45:13.28+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

While reviewing your support request, we were unable to locate an active entitlement/subscription associated with the account, which is preventing us from creating the DFM case.

Could you please confirm whether you have an active subscription linked to the affected resource? If available, kindly share the Subscription ID or any relevant subscription details so that we can proceed further with the case creation.

We look forward to your response.

Thanks & Regards,
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-09T08:31:04.5566667+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-11T03:38:48.38+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Andrei Josephsen 0 Reputation points

2026-09-11T08:13:22.95+00:00

"The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions." -

I wish there was some way for users to pick the algorithm - and the preferred would probably be to use the same Murmur2 hashing as Kafka .

Republishing every event, as answer suggest works, but as it ads latency and cost, I would prefer the hashing algorithm to be consistent.

But - answer would be accepted - but as it is just a comment - I don't think I have any way to accept it (technically)
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-12T09:22:28.8866667+00:00

Hi @Andrei Josephsen ,

Since the partition key hash algorithm is an internal implementation detail and cannot currently be configured, there is no supported method to enforce Kafka Murmur2 partition mapping within the service.

The following mitigation options may be considered:

Continue using the existing partition key behavior if cross-platform partition consistency is not a strict requirement.

Republish events through an intermediate processing layer to ensure consistent partition assignment when interoperability with Kafka producers is required.

Standardize message production through a single producer path to minimize differences in partition routing behavior across services.

Submit a product feedback request to the engineering team requesting support for a configurable partition hashing algorithm or Kafka-compatible Murmur2 hashing.

Review application design to determine whether strict partition alignment is required or whether event ordering can be maintained through alternative mechanisms.

Limitation

At present, the partition key hash algorithm is not exposed as a customer-configurable setting. Therefore, different services may map the same partition key value to different physical partitions, and this behavior is expected.

Recommended Resolution

If deterministic partition mapping across Kafka and Azure services is a business requirement, republishing or implementing a custom routing layer remains the most practical workaround. However, this approach introduces additional latency and operational cost. A product enhancement request for configurable hashing algorithms would be the preferred long-term solution.

Thanks & Regards,
Sivasankar.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-16T06:13:45.46+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 170 Reputation points Microsoft External Staff Moderator

2026-09-17T04:43:51.5933333+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.