Spark: Check Table Exists Quickly
This code snippet checks for table existence really fast:
def table_exists(spark: SparkSession, table_name: str) -> bool:
# a more logical way would be to use spark catalog, however it's really slow
from pyspark.sql.utils import AnalysisException
try:
spark.table(table_name)
except AnalysisException:
return False
return True
To contact me, send an email anytime or leave a comment below.