An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
inferschema error on parquet files, mergeSchema is time consuming
I stored the parequet files in ADLS Gen2. Most time I read the file with no problem, but some time the collected parquet files triggered the "can not inferSchema" errors. I googled the error, most likely it was caused by an empty parquet files. One suggested fix is to turn on the option "mergeSchema" , like:
parquetFile = spark.read.option("ignoreCorruptedFiles", "true").option("mergeSchema","true").parquet(*fileForModels)
The option fixed the error, but the runtime is substantially longer (tripled or more). My questions are:
- how to find the problematic parquet files in the ADLS Gen2? I checked each directory, the only zero size file is "_SUCCESS", but it was present in each directory, and I don't think it is the problem
- Is there any better fix than "mergeSchema" option?
Azure Data Lake Storage
1 answer
Sort by: Newest
-
Deleted
This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Comments have been turned off. Learn more