Data links KW 4

some useful & interesting links

def cacheAs(df:org.apache.spark.sql.DataFrame, name:String level:org.apache.spark.storage.StorageLevel) :org.apache.spark.sql.DataFrame = {
try spark.catalog.uncacheTable(name)
  catch { case _: org.apache.spark.sql.AnalysisException => () }
  df.createOrReplaceTempView (name)
  spark.catalog.cacheTable(name, Lever)!
return df
}

which gives cached RDDs nicer names and thus eases debbugging

Avatar
Georg Heiler
PhD candidate & data scientist

My research interests include large geo-spatial time and network data analytics.

comments powered by Disqus