Workspace Transform Data Collection Rule for Log Analytics Workspace on Diagnosic Settings on (e.g. Azure FrontDoor)

Question

Workspace Transform Data Collection Rule for Log Analytics Workspace on Diagnosic Settings on (e.g. Azure FrontDoor)

steinfin 0

I'm trying to setup a Diagnostic Setting on a Azure Front Door resource to emit logs to a Log Analytics Workspace. However the loglines may contain PII, and we host in Europe, so I want to cover my bases to be GDPR compliant - so I need to transform the lines on Ingestion.

AFAICT, the workspace transforms Data Collection Rules, can provide this functionality. I manage to transform log lines from Blob Storage which uses a dedicated table, but for Azure Front Door the log lines go to AzureDiagnostic Table, and the DCR rules applied for that Data Flow doesn't seem to apply.

Can one not perform data minimization on Azure FrontDoor resource diagnostic settings logs at all?
- If not; is there any other mechanism than diagnostic settings on front door that allow us to log url redirection and caching?

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Hello @steinfin - Front Door writes its logs to AzureDiagnostics, as you found, and AzureDiagnostics is not a table that supports transformations. That is the whole problem. A workspace transformation DCR only applies to tables that have transformation support, so yours is being silently ignored. Nothing in it is misconfigured, it just cannot fire against that table.

It is also why your Blob Storage rule works. Those logs land in StorageBlobLogs, which does support them. The two rows from the Azure Monitor Logs table feature support page:

StorageBlobLogs carries the DCR indicator, AzureDiagnostics does not. The data collection transformations documentation sets the rule: supported tables are "Any Azure table listed in Logs table feature support with the indicator." Front Door has no resource-specific table either, so there is nowhere else to send the data.

So for your first question: no, not through a workspace transformation DCR.

For your second question: yes, but not through the diagnostic setting alone. Point it at an event hub instead of Log Analytics, which is what the docs suggest event hubs for, "integrating with other security information and event management (SIEM) tools or external data stores". Minimize in your own consumer, then push the reduced payload into a custom table with the Logs Ingestion API. You supply the DCR there, and that DCR can carry a transformation: "You can also use the transformation to filter source data and perform any other calculations or conversions."

More moving parts, but the minimization happens before anything reaches the workspace, which is where you want it for GDPR. And you keep the FrontDoor Access Log category, so you do not lose the URL and caching detail you are collecting this for.

Help make this community better for everyone: if this answer resolved your issue, please accept it or leave an upvote. If not, share more details in a comment so we can continue the discussion and find the right solution.