A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
Do not deduplicate these rows by exported values; retain each physical CSV occurrence, replace previously loaded report partitions rather than appending regenerated reports, and open an Azure billing support request because the public documentation does not explain or guarantee distinguishability for fully identical records.
What the documentation establishes
- Cost Details files have no native per-record ID. Microsoft says fields marked
¹can be combined to create an ID, while also stating that every record in the file should be considered unique.^1^ - A resource or meter can legitimately have multiple records on one day. However, the documented example attributes this to differences in
AdditionalInfo; it does not address records identical across every exported field.^2^ - Amortized Cost reallocates reservation and savings-plan costs to benefiting resources. That explains why amortized data can contain multiple charge allocations, but it does not establish that two fully identical exported rows are expected or provide a hidden identifier consumers can use.^3^
Consequently, the available public contract does not answer whether your pairs represent separate underlying usage or a reporting defect. It also exposes no additional supported field that can distinguish them. The documented identity guidance needs clarification if rows can legitimately collide across all exported values.
Your reconciliation result is operationally decisive: because Azure’s query and report totals include both occurrences, value-based deduplication would understate Azure-reported quantity and cost.
Idempotent ingestion design
Treat each generated report as a snapshot for its requested scope, metric, schema version, and time range, not as an append-only event stream:
- Store the original file and report metadata unchanged.
- Assign each physical row an ingestion-local ordinal, such as:
- report/run identifier;
- source file or partition name;
- row number within that file.
- Keep a hash of the decoded row as a fingerprint for diagnostics, but do not use that hash as a uniqueness constraint. Identical hashes can have multiplicity greater than one.
- Load the new report into staging.
- Validate its totals and row counts against the downloaded source.
- In one transaction, replace the previously loaded partition for the exact same scope, metric, schema, and covered period.
- Publish the replacement only after validation succeeds.
For example:
begin transaction;
delete from cost_detail
where billing_scope = @scope
and metric = 'AmortizedCost'
and usage_date between @start_date and @end_date;
insert into cost_detail (
billing_scope, metric, report_id, source_file, row_ordinal,
row_fingerprint, usage_date, quantity, cost, raw_row
)
select
billing_scope, metric, report_id, source_file, row_ordinal,
row_fingerprint, usage_date, quantity, cost, raw_row
from cost_detail_stage;
commit;
The key point is that row_ordinal distinguishes physical occurrences only within a particular report snapshot. It must not be treated as a stable Azure identity across regenerated reports.
For scheduled Cost Management Exports, Azure provides an Overwrite data option, enabled by default for new exports; daily exports then replace the previous day’s exported file with updated data.^4^ This is the clearest documented replacement model. It does not create a row identity, but it can simplify downstream idempotency: ingest each overwritten path as a replacement snapshot rather than appending it.
Replacement is also important because charges in the current billing period can be rerated until the invoice closes.^5^ Avoid merging regenerated reports solely by row hash: that cannot distinguish a legitimate multiplicity change from a rerating correction.
Private investigation and documentation clarification
Open Azure portal → Help + support → Create a support request, selecting the subscription associated with the billing profile and the closest Billing/Cost Management problem classification. Creating the request requires Owner, Contributor, Support Request Contributor, or equivalent Microsoft.Support/* permission at subscription scope.^6^
Attach a compressed diagnostic package containing:
- Cost Details operation/report identifiers and generation timestamps.
- Billing-profile scope and requested date range.
- Metric and schema version.
- Source file/partition names and row numbers.
- SHA-256 hashes of the original files and affected decoded rows.
- A redacted extract preserving duplicate multiplicity.
- Expected versus actual row counts, quantities, and costs.
- Corresponding Cost Management query request dimensions and returned totals.
- Confirmation that a fresh report reproduced the same pairs.
Ask support to confirm specifically:
- Whether each occurrence maps to a distinct internal rated-usage or amortization record.
- Whether an internal distinction was omitted from the public schema.
- Whether the output violates the documented
¹-field identity guidance. - Whether consumers should use report-level replacement as the supported idempotency boundary.
Until Microsoft makes that determination, preserving both physical occurrences and reconciling their aggregate is the only approach consistent with the published Cost Details totals and the instruction to consider every record unique.
References