Data links KW 4

Jan 27, 2019·
Georg Heiler
Georg Heiler
· 1 min read
def cacheAs(df:org.apache.spark.sql.DataFrame, name:String level:org.apache.spark.storage.StorageLevel) :org.apache.spark.sql.DataFrame = {
try spark.catalog.uncacheTable(name)
  catch { case _: org.apache.spark.sql.AnalysisException => () }
  df.createOrReplaceTempView (name)
  spark.catalog.cacheTable(name, Lever)!
return df
}

which gives cached RDDs nicer names and thus eases debbugging

Georg Heiler
Authors
senior data expert
My research interests include large geo-spatial time and network data analytics.