ASA parallel JOIN drops matches when the two Event Hubs inputs use different partition-key hashing

Question

ASA parallel JOIN drops matches when the two Event Hubs inputs use different partition-key hashing

Andrei Josephsen 40

ASA job (compat 1.2) joins two Event Hubs on EntityId (both 4 partitions) into a third hub. With the input Partition key column declared (embarrassingly-parallel), the JOIN silently loses ~78% of matches — no errors. Measured: parallel ≈ 22% match, non-parallel ≈ 100%.

Cause: the two inputs are produced by clients that hash the key differently, so the same EntityId lands on different partitions: EventHubBufferedProducerClient uses client-side lookup3, EventHubProducerClient/gateway uses another hash, and ASA output uses a third.

Q: Can I force all publishers (buffered producer, regular producer, and ASA output) to use the same partition-key hashing function, so the same EntityId always lands on the same partition and a parallel JOIN matches?

SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-25T12:55:26.5066667+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

To investigate this issue further, I have sent you a private message. Could you please provide the following details through the private message?

Error or failure screenshots

Your email address

Time zone

Your availability for a Microsoft Teams meeting

These details will help us perform a deeper investigation and engage the appropriate engineering team if required.

Once you've shared the requested information, we will continue our analysis and provide further guidance.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-26T08:47:01.2333333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
Andrei Josephsen 40 Reputation points

2026-08-26T11:00:46.51+00:00

The reason is fairly clear ... events with a partitionID get routed to different partitions depending on how they were published .. example with four partitions

event with PartitionKey: 8898d388-f719-461f-b26d-2035fbb732f8ends on partition 1 using ASA

ends on partition 2 using sdk EventHub BufferedProducerClient

This basically ruins out combining ASA events with BufferedProducerClient events for high throughput scenarios
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-31T09:55:13.7033333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-02T06:45:13.28+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

While reviewing your support request, we were unable to locate an active entitlement/subscription associated with the account, which is preventing us from creating the DFM case.

Could you please confirm whether you have an active subscription linked to the affected resource? If available, kindly share the Subscription ID or any relevant subscription details so that we can proceed further with the case creation.

We look forward to your response.

Thanks & Regards,
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-02T07:13:24.33+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

Based on the current Azure Event Hubs and Azure Stream Analytics design, it is not possible to force EventHubProducerClient, EventHubBufferedProducerClient, and Azure Stream Analytics outputs to use the same partition-key hashing function.

The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions.

For Azure Stream Analytics parallel JOINs, the engine assumes that records with the same partition key are co-located in corresponding partitions across all inputs. If the upstream producers use different partitioning behaviors and the same EntityId ends up in different partitions, matching events may never be processed together, causing the reduced match rate you observed.

As a workaround, consider one of the following approaches:

• Disable the partition-key optimization and run the JOIN without declaring the Partition key column. This typically restores correctness but may reduce throughput.

• Ensure both input streams are produced through a single consistent publishing path so that identical EntityId values are partitioned consistently before reaching Stream Analytics.

• Repartition or normalize the streams upstream (for example, through an intermediary processing layer) before they are consumed by the ASA job.

Therefore, there is currently no supported mechanism to configure a common partition-key hashing function across EventHubBufferedProducerClient, EventHubProducerClient, and Azure Stream Analytics output destinations.

If maintaining an embarrassingly parallel JOIN is a requirement, the recommended approach is to ensure partition alignment upstream rather than relying on configurable hashing behavior.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-09T08:31:04.5566667+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Andrei Josephsen 40 Reputation points

2026-09-11T08:13:22.95+00:00

"The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions." -

I wish there was some way for users to pick the algorithm - and the preferred would probably be to use the same Murmur2 hashing as Kafka .

Republishing every event, as answer suggest works, but as it ads latency and cost, I would prefer the hashing algorithm to be consistent.

But - answer would be accepted - but as it is just a comment - I don't think I have any way to accept it (technically)
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-16T06:13:45.46+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-26T04:25:20.3166667+00:00

Hi Andrei,

Thank you for the detailed information and testing you have provided regarding the Azure Stream Analytics parallel JOIN behavior.

Based on the scenario, the issue is related to partition alignment between the two Event Hubs inputs when the ASA job is configured for partition-parallel processing.

Issue

The ASA job joins two Event Hubs inputs using EntityId. When the Partition key column is configured, you are observing approximately 22% matching records, while removing the parallel partition configuration results in approximately 100% matches.

Your testing also shows that the same EntityId can be placed in different physical Event Hub partitions depending on the publishing path.

Technical Analysis

For Azure Stream Analytics to process a query in parallel, the partitioning needs to remain aligned across the inputs and query processing. For a JOIN, records associated with the same JOIN key need compatible partitioning so that the corresponding records can be processed together.

When the same EntityId from the two inputs is mapped to different partitions, the existing Event Hub partition layout cannot safely be treated as aligned for the parallel JOIN. This explains why removing the partition-key optimization restores the expected matching behavior in your testing.

Regarding forcing EventHubBufferedProducerClient, EventHubProducerClient, and ASA output to use one specific hashing algorithm, we should not rely on a particular physical hash implementation as the solution unless that behavior is explicitly supported and documented.

Recommended Mitigation

For immediate correctness, remove the partition-key optimization and run the JOIN without assuming that the existing Event Hub partitions are aligned.

For correctness with scalability, repartition or normalize both input streams using the actual JOIN key, EntityId, before performing the parallel JOIN.

If an intermediate processing stage is acceptable, both streams can be normalized/repartitioned before being consumed by the final ASA JOIN. The resulting streams should use the same logical partition key and compatible partition counts.

If possible, standardize the publishing architecture to reduce partition-routing differences. However, this should not be treated as a guarantee of a particular common hashing algorithm unless explicitly documented.

Therefore, the recommended long-term design is to align/repartition both streams by EntityId before the parallel JOIN, rather than depending on the physical partition selected by different publishers.

I hope this clarifies the observed behavior and provides a path to maintain both correctness and scalability.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-30T01:06:26.7533333+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.

Answer accepted by question author

0 additional answers

Your answer

SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-25T12:55:26.5066667+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

To investigate this issue further, I have sent you a private message. Could you please provide the following details through the private message?

Error or failure screenshots

Your email address

Time zone

Your availability for a Microsoft Teams meeting

These details will help us perform a deeper investigation and engage the appropriate engineering team if required.

Once you've shared the requested information, we will continue our analysis and provide further guidance.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-26T08:47:01.2333333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
Andrei Josephsen 40 Reputation points

2026-08-26T11:00:46.51+00:00

The reason is fairly clear ... events with a partitionID get routed to different partitions depending on how they were published .. example with four partitions

event with PartitionKey: 8898d388-f719-461f-b26d-2035fbb732f8ends on partition 1 using ASA

ends on partition 2 using sdk EventHub BufferedProducerClient

This basically ruins out combining ASA events with BufferedProducerClient events for high throughput scenarios
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-08-31T09:55:13.7033333+00:00

Hi @Andrei Josephsen ,

I hope you are doing well.

This is a gentle follow-up regarding the information requested in my previous message Could you please share the requested details when you have a chance.

This will help us continue our investigation and assist you further.

Thank you for your time and cooperation.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-02T06:45:13.28+00:00

Hi @Andrei Josephsen ,

Thank you for reaching out to the Microsoft Q&A forum.

While reviewing your support request, we were unable to locate an active entitlement/subscription associated with the account, which is preventing us from creating the DFM case.

Could you please confirm whether you have an active subscription linked to the affected resource? If available, kindly share the Subscription ID or any relevant subscription details so that we can proceed further with the case creation.

We look forward to your response.

Thanks & Regards,
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-09T08:31:04.5566667+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Andrei Josephsen 40 Reputation points

2026-09-11T08:13:22.95+00:00

"The partition key hash algorithm used to map events to partitions is an internal implementation detail and is not exposed as a configurable setting. As a result, different producers or services may route the same partition key value to different physical partitions." -

I wish there was some way for users to pick the algorithm - and the preferred would probably be to use the same Murmur2 hashing as Kafka .

Republishing every event, as answer suggest works, but as it ads latency and cost, I would prefer the hashing algorithm to be consistent.

But - answer would be accepted - but as it is just a comment - I don't think I have any way to accept it (technically)
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-16T06:13:45.46+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator

2026-09-30T01:06:26.7533333+00:00

Hi @Andrei Josephsen ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.

Answer 1

kagiyama yutaka 5,575

I think the ASA-side fix is to repartition both streams by EntityId with the same partition count before the JOIN, rather than trying to align partition mapping across producers.

0 comments

Answer 2

Hi @Andrei Josephsen ,

Since the partition key hash algorithm is an internal implementation detail and cannot currently be configured, there is no supported method to enforce Kafka Murmur2 partition mapping within the service.

The following mitigation options may be considered:

Continue using the existing partition key behavior if cross-platform partition consistency is not a strict requirement.
Republish events through an intermediate processing layer to ensure consistent partition assignment when interoperability with Kafka producers is required.
Standardize message production through a single producer path to minimize differences in partition routing behavior across services.
Submit a product feedback request to the engineering team requesting support for a configurable partition hashing algorithm or Kafka-compatible Murmur2 hashing.
Review application design to determine whether strict partition alignment is required or whether event ordering can be maintained through alternative mechanisms.

Limitation

At present, the partition key hash algorithm is not exposed as a customer-configurable setting. Therefore, different services may map the same partition key value to different physical partitions, and this behavior is expected.

Recommended Resolution

If deterministic partition mapping across Kafka and Azure services is a business requirement, republishing or implementing a custom routing layer remains the most practical workaround. However, this approach introduces additional latency and operational cost. A product enhancement request for configurable hashing algorithms would be the preferred long-term solution.

Thanks & Regards,
Sivasankar.