Spark Union in Pythonic Way
To do union in more pythonic way with variable args i.e. union(df1, df2, df3) here is a snippet for a helper function:
def union_all(*df: DataFrame) -> DataFrame or None:
"""
Unions all dataframes. Null dataframes are skipped.
:param df:
:return:
"""
rdf = None
for idf in df:
if not idf:
continue
rdf = rdf.union(idf) if rdf else idf
return rdf
To contact me, send an email anytime or leave a comment below.