help you fix your problem The result is correct. You are doing a transformation only as select is a transformation. If you save it in a parquet file or a csv file you will see the result and can confirm that the column is already selected.
val selecteddf = customerdf.select("name")
Error Running Cassandra from Spark in Java - NoClassDefFoundError at org.apache.spark.sql.catalyst
How to get execution DAG from spark web UI after job has finished running, when I am running spark on YARN?
By : Heena
Date : March 29 2020, 07:55 AM
To fix the issue you can do Running only history-server is not sufficient to get execution DAG of previous jobs. You need specify the jobs to store the events logs of all previous jobs. Run Spark history server by ./sbin/start-history-server.sh