How to identify corrupt Parquet Files

Kieran Wood 111 Reputation points
2023-10-23T10:31:18.0433333+00:00

How can I detect corrupt Parquet files? I recently ran a Synapse Pipeline/Data Flow, and when I previewed the data in the source data flow component, I received the following error: "Synapse Pipeline ADF at Source Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 204) (vm-70b97998 executor 1): org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)"

I think the error indicates one or more corrupt Parquet files. The data source container/folder contains around 300 Parquet files. How do I iterate through these files and identify the corrupt Parquet files?

Azure Data Lake Storage
Azure Data Lake Storage

An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    2 deleted comments

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.