Spark DataFrame: Display Last Columns
Working on very wide dataframes is painful, to display only last n columns you can use:
def get_last_cols(df: DataFrame, n: int = 3):
"""Gets last columns from a DataFrame (3 by default)
:param df:
:param n:
:return:
"""
cols = df.schema.fieldNames()
last_cols = cols[-n:]
return df.select(*last_cols)
usage:
display(get_last_cols(df, 5))
To contact me, send an email anytime or leave a comment below.