![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 74%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELH%3C/text%3E%3C/svg%3E)
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
Hi @Lam, Hoanh
The stack trace is useful because the failure is occurring specifically at df.write.mode("append").format("delta").saveAsTable(<TABLE>) inside your foreachBatch callback.
Normally, when <TABLE> already exists and resolves to a compatible Delta table, mode("append").saveAsTable() should append to it rather than attempt to create it. So don't drop or recreate the table based on this exception.
First, check whether the table name resolves consistently inside the streaming job. If you're using an unqualified name such as:
saveAsTable("my_table")
change the test to the fully qualified Unity Catalog name:
target = "catalog.schema.my_table"
df.write \
.format("delta") \
.mode("append") \
.saveAsTable(target)
Then, immediately before the write inside process_batch, log what Spark sees:
target = "catalog.schema.my_table"
print("catalog =", spark.catalog.currentCatalog())
print("database =", spark.catalog.currentDatabase())
print("tableExists =", spark.catalog.tableExists(target))
spark.sql(f"DESCRIBE DETAIL {target}").show(truncate=False)
df.write \
.format("delta") \
.mode("append") \
.saveAsTable(target)
This matters because your exception isn't a typical schema-mismatch error. Databricks is reporting:
[TABLE_OR_VIEW_ALREADY_EXISTS]
Cannot create table or view because it already exists.
SQLSTATE: 42P07
Also, check the table history around the exact time the micro-batch failed:
DESCRIBE HISTORY catalog.schema.my_table;
Look for concurrent DDL or lifecycle operations such as CREATE, CREATE OR REPLACE, DROP, schema changes, or another job/pipeline manipulating the same target. If another process changed the catalog object while the stream was running, that would be important evidence.
Also confirm what object Spark believes <TABLE> actually is:
DESCRIBE EXTENDED catalog.schema.my_table;
In particular, verify that it is the expected Delta table, not a view, streaming table, materialized view, or an object whose ownership/lifecycle is managed by another pipeline.
Don't add IF NOT EXISTS, OR REPLACE, or automatically drop the table just because those suggestions appear in the generic exception text. Those are remedies for explicit table-creation operations and don't explain why an established append stream unexpectedly entered a create path.
Since you mentioned that the streaming job had been running successfully for some time before one batch failed, could you also provide:
- Databricks Runtime version
- Dedicated/shared/serverless compute
- Unity Catalog or Hive metastore
- Whether <TABLE> is fully qualified
- Whether any other job/pipeline writes to or modifies the same table
If tableExists() returns True, DESCRIBE DETAIL identifies the expected Delta table, there was no concurrent DDL, and the same saveAsTable(..., mode="append") code intermittently enters the create path after previously succeeding, then capture the failing job/run ID, cluster/compute ID, UTC timestamp, full exception, target table name, and Delta history and raise it with Azure Databricks Support rather than work around it by recreating the table.
One more point: because this is inside foreachBatch, make sure the batch write is idempotent before manually replaying/restarting failed batches. Databricks notes that foreachBatch provides at-least-once write guarantees by default, so a restarted/replayed batch needs appropriate handling if duplicate writes would be a problem.
References:
Databricks - Use foreachBatch to write to arbitrary data sinks
Databricks - Delta table history
Help make this community better for everyone: If this answer helped or resolved your issue, please accept it or upvote it. If not, share more details in a comment so we can continue the discussion and find the right solution. Thank you.