What happened to my thread?

Discussion in 'Big Data and Analytics' started by Vaughn Shideler, May 7, 2017.

  1. Vaughn Shideler

    Joined:
    Mar 13, 2017
    Messages:
    7
    Likes Received:
    0
    A few days ago I posted a thread about the k-means algorithm assignment and it disappeared shortly thereafter. Did someone delete it? If so, why?
     
    #1
  2. Megha_42

    Megha_42 Well-Known Member
    Simplilearn Support

    Joined:
    Dec 15, 2016
    Messages:
    206
    Likes Received:
    8
    Hi Vaughn,

    Thank you for reaching out.
    We never delete any threads unless they are Spam or contains abusive or unacceptable content. We always answer all our participant queries. However, there might have been some errors during the time of creation which might have caused problems.

    Could you kindly put your query regarding the k-means algorithm again, on this thread, so we can answer it and help you complete the Project?

    We are truly sorry for the inconvenience in this regard.

    Looking forward to hearing back from you

    Warm regards
    Megha
     
    #2
  3. Vaughn Shideler

    Joined:
    Mar 13, 2017
    Messages:
    7
    Likes Received:
    0
    As mentioned, the issues relate to doing project 2 - the "Loudacre Mobile" assignment using a k-means algorithm.

    I realize that you probably can't "give away" direct answers to the problems in the assignment, but I'm hoping that you can at least point me in the right direction. It's my belief that the supplied course materials do not provide enough information to complete the tasks.

    The first issue is that in the "Project 2_dataset" file, there is a timestamp field that uses a combination of dashes and colons. I don't know what data type to use to process these. For example, if I do:

    case class Mobile(time:String, name:String, id:String, lat:Int, long:Int)

    and then map it with:

    val mobilerdd = input_split.map(x => Mobile(x(0), x(1), x(2), x(3).toInt, x(4).toInt))

    I create a data frame:

    val mobileDF = mobilerdd.toDF()

    But when I try to do a "mobileDF.show" I get:

    ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 2)java.lang.NumberFormatException: For input string: "33.6894754264"

    Obviously it's not accepting the latitude and longitude as an integer, but I don't know what else to process it as.

    The next question is how to process the algorithms in spark. The only thing that we were given in the course materials is a list of the k-means parameters, such as maxiterations, initializationmode, etc. This isn't enough, though. I found something on Apache's documentation for clustering:

    http://spark.apache.org/docs/latest/ml-clustering.html#k-means

    So it seems that Spark uses commands such as

    val WSSSE = model.computeCost(dataset)

    to process some of the calculations, but this still doesn't tell us how to apply it to the current project.
     
    #3
  4. Vaughn Shideler

    Joined:
    Mar 13, 2017
    Messages:
    7
    Likes Received:
    0
    Is there any update?
     
    #4
  5. Vaughn Shideler

    Joined:
    Mar 13, 2017
    Messages:
    7
    Likes Received:
    0
    So none of the support staff can answer my questions? This proves to me that simplilearn's big data course is insufficient to do the kind of work needed in Hadoop and Spark. Very disappointing. I'm willing to bet that the only people who have successfully completed project 2 (if any) have prior experience with either programming or Spark.
     
    #5
  6. Karthik Shivana

    Karthik Shivana Moderator
    Simplilearn Support

    Joined:
    Apr 1, 2016
    Messages:
    638
    Likes Received:
    25

    Hi Vaughn Shideler,

    Apologize for the delay in response, please find the below details,

    The data is not loaded properly, so that data-frame is not created properly.

    Please look into this
    ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 2)java.lang.NumberFormatException: For input string: "33.6894754264"

    The values are not in integer its in double.

    Please let me know, if you still find any issues on the same.
     
    #6

Share This Page