How do I route and append iot hub messages into a csv stored in blob storage

Question

How do I route and append iot hub messages into a csv stored in blob storage

Sydney 31

Hi experts
What would you recommend as a method to send messages from iot hub and append them into an existing csv file in Blob Storage?

Thanks in advance!

QuantumCache 20,306 Reputation points Moderator

2020-07-08T04:15:32.88+00:00

Hello @Sydney , Just checking back, could you please let us know if you need further help in this matter.

2 answers

Your answer

QuantumCache 20,306 Reputation points Moderator

2020-07-08T04:15:32.88+00:00

Hello @Sydney , Just checking back, could you please let us know if you need further help in this matter.

Answer 1

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

2 deleted comments

Comments have been turned off. Learn more

Answer 2

AshokPeddakotla-MSFT 35,346 Moderator

@Sydney Welcome to Q&A forum.

Please elaborate your query bit more to understand better.
Have you already configured the routing? Are you following any documentation for the steps?

The data can be written to blob storage in either the Apache Avro format, which is the default, or JSON (preview).

Please note that, IoT Hub supports writing data to Azure Storage in the Apache Avro format as well as in JSON format. The default is AVRO. The encoding format can be only set when the blob storage endpoint is configured. The format cannot be edited for an existing endpoint. When using JSON encoding, you must set the contentType to application/json and contentEncoding to UTF-8 in the message system properties. Both of these values are case-insensitive. If the content encoding is not set, then IoT Hub will write the messages in base 64 encoded format.

I would recommend you to check the Configure IoT Hub message routing and Routing messages to blob storage and see if it helps to understand the routing endpoints.

Sydney 31 Reputation points

2020-06-30T15:35:54.7+00:00

Hi Ashok
Thank you for your answer.

Let me try to elaborate - I have downstream consumers that are expecting the telemetry data to be in a csv file in Blob storage, and they will consume it from there.

My question was - what would be a good strategy to get data from iot hub to a csv file in blob ? I know iot hub will not write it to a csv directly, so the question really is should I use Azure functions? Should I use Stream Analytics? Should I use an Event Hub? What's a good practice? what should I consider? How would you suggest I develop this?

Hope this clarifies.
Sydney 31 Reputation points

2020-06-30T15:37:09.553+00:00

And to further clarify - I need to APPEND to a file. i.e. not create separate files in blob, as the json routing to blob would likely do. Thanks