Import CSV file to Scala DF

Discussion in 'Big Data and Analytics' started by ADITYA SHIVAM, Jun 25, 2017.

  1. ADITYA SHIVAM

    ADITYA SHIVAM Member
    Alumni

    Joined:
    Apr 8, 2017
    Messages:
    12
    Likes Received:
    1
    I need to know, how can we upload csv file in scala data frame, This was not covered in our BDHS classes and unfortunately project is on the same topic.

    Appreciate quick responses here.

    Thanks

    Aditya
     
    #1
  2. Megha_42

    Megha_42 Well-Known Member
    Simplilearn Support

    Joined:
    Dec 15, 2016
    Messages:
    206
    Likes Received:
    9
    Hi Aditya,

    You will need to include a databricks package while you start the Spark shell.

    Here's how you do it,

    spark-shell --packages com.databricks:spark-csv_2.10:1.5.0

    Here is a sample command
    val df = sqlContext.read.format("com.databricks.spark.csv").option("header","true").option("inferSchema", "true").load("filename.csv");

    Before you run this, please ensure that the dataset is free of junk symbols and the quotes are consistent.

    All the very best!
     
    #2

Share This Page