How to automate Downloading Message Trace logs on a Daily Basis when the entries are more than 10,000,000 per day.

Hi, @Newton Francis Braganza

To automate the download of Message Trace logs for more than 10,000,000 records per day, you can try the following suggestions:

1. Considering that the entries far exceed the limit, you need to split the query into smaller time intervals to ensure that this limit is not exceeded.
2. For larger datasets, you can use Start-HistoricalSearch and Get-HistoricalSearch to search for message data.
3. Create a PowerShell script to automate this process.

Please understand that Exchange Online tag is not focused on scripts at the moment, in order to better solve your problem, I will add PowerShell tag for you.

Based on my personal experience, the following script is for reference.

# Define the start and end date for the query
$startDate = (Get-Date).AddDays(-1).ToString("yyyy-MM-ddTHH:mm:ssZ")
$endDate = (Get-Date).ToString("yyyy-MM-ddTHH:mm:ssZ")
 
# Define the interval in hours
$intervalHours = 1
 
# Loop through the intervals and retrieve the message trace logs
for ($i = 0; $i -lt 24; $i += $intervalHours) {
    $intervalStart = (Get-Date).AddHours(-$i - $intervalHours).ToString("yyyy-MM-ddTHH:mm:ssZ")
    $intervalEnd = (Get-Date).AddHours(-$i).ToString("yyyy-MM-ddTHH:mm:ssZ")
 
    # Retrieve the message trace logs for the interval
    $messageTrace = Get-MessageTrace -StartDate $intervalStart -EndDate $intervalEnd -PageSize 100000
 
    # Save the logs to a file
    $fileName = "MessageTrace_$($intervalStart)_to_$($intervalEnd).csv"
    $messageTrace | Export-Csv -Path $fileName -NoTypeInformation
}
 
# Combine all the CSV files into a single file
$combinedFileName = "Combined_MessageTrace_$(Get-Date -Format 'yyyyMMdd').csv"
Get-ChildItem -Path . -Filter "MessageTrace_*.csv" | ForEach-Object {
    Import-Csv -Path $_.FullName
} | Export-Csv -Path $combinedFileName -NoTypeInformation

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

As @Anonymous pointed out, you'll have to retrieve the information in smaller chunks. His code chops up the data into chunks of one hour each. You may need to shorten that interval and do the data retrieval in two (nested) loops, not just one-- and grab the date of the last message traced to be used to get the next batch.

Here's an example

https://robdy.io/message-trace-wrapper/#:~:text=Documentation%20for%20Get-MessageTrace%20states%20about%20the%20limitations:%20By,it%20up%20using%20smaller%20StartDate%20and%20EndDate%20intervals.

You may need to combine the code in the link with the code @Anonymous provided, but what you want to accomplish is doable -- it just takes you using the approach that works in your situation.