inferschema error on parquet files, mergeSchema is time consuming

P, John 220 Reputation points
2024-02-21T00:57:05.95+00:00

I stored the parequet files in ADLS Gen2. Most time I read the file with no problem, but some time the collected parquet files triggered the "can not inferSchema" errors. I googled the error, most likely it was caused by an empty parquet files. One suggested fix is to turn on the option "mergeSchema" , like:

parquetFile = spark.read.option("ignoreCorruptedFiles", "true").option("mergeSchema","true").parquet(*fileForModels)

The option fixed the error, but the runtime is substantially longer (tripled or more). My questions are:

  1. how to find the problematic parquet files in the ADLS Gen2? I checked each directory, the only zero size file is "_SUCCESS", but it was present in each directory, and I don't think it is the problem
  2. Is there any better fix than "mergeSchema" option?
Azure Data Lake Storage
Azure Data Lake Storage

An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.

0 comments No comments

1 answer

Sort by: Newest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.


    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.