Big Data Hadoop and Spark Developer| Rakesh | March 23rd - April 28th

Discussion in 'Big Data and Analytics' started by Neha_Pandey, Mar 26, 2019.

  1. Neha_Pandey

    Neha_Pandey Well-Known Member
    Simplilearn Support Alumni

    Joined:
    Jun 7, 2018
    Messages:
    95
    Likes Received:
    0
    Hi Learners,

    Kindly post your queries below.

    Happy Learning,

    Regards,
    Neha Pandey
     
    #1
  2. Sai Chand pasula

    Sai Chand pasula New Member

    Joined:
    Mar 14, 2019
    Messages:
    1
    Likes Received:
    0
    Can anyone able to create dataset in your webconsole?
     
    #2
  3. Rakesh_236

    Rakesh_236 Active Member

    Joined:
    Dec 27, 2018
    Messages:
    22
    Likes Received:
    2
    Hello everyone,

    I have uploaded following to google drive.
    1) SQOOP and Map reduce command line interface command file covering all that I demoed
    2) 2 assignments on map reduce for those who are interested
    3) you will already have HDFS/MYSQL assignment from last week so please complete those
    4) Java code for map reduce with needed input files:
    -- Wordcount
    -- reduce side join
    -- composite join

    Happy Learning!
    Regards
    Rakesh Srivastva
     
    #3
    _8709 likes this.
  4. Jiaming Zi

    Jiaming Zi Member

    Joined:
    Mar 29, 2019
    Messages:
    2
    Likes Received:
    0
    Hi Rakesh,

    I cannot find the "team_statistics" folder as mentioned in the MapReduce assignment document. Can you tell me exactly where it it? Thanks
     
    #4
  5. Boopathy_1

    Boopathy_1 Member

    Joined:
    Feb 8, 2019
    Messages:
    4
    Likes Received:
    0
    Hi Rakesh,

    I unable to setup the sandbox becoz of system limitations.
    But i setup the eclipse and exported the jar file

    But now
    how to execute it??

    Kindly help
     
    #5
  6. Rakesh_236

    Rakesh_236 Active Member

    Joined:
    Dec 27, 2018
    Messages:
    22
    Likes Received:
    2
    Hi,

    Steps:
    1) Copy the jar file to LMS
    2) Run below command from LMS Linux window:
    $hadoop jar <give your jar file name> <java package.class name> <Input file path of HDFS> <Output folder path of HDFS>

    Let me know if this helps. I will check again tomorrow evening.

    Regards
    Rakesh
     
    #6
  7. Rakesh_236

    Rakesh_236 Active Member

    Joined:
    Dec 27, 2018
    Messages:
    22
    Likes Received:
    2
    Hi,

    I now remember, that team_statistics was more of a reference solution to solve retail problem. I have now added the player_statistics,csv file and also the java code to calculate team_statistics in the google drive. please refer the same.

    Regards
    Rakesh
     
    #7
  8. Boopathy_1

    Boopathy_1 Member

    Joined:
    Feb 8, 2019
    Messages:
    4
    Likes Received:
    0
    Hi Rakesh,

    Titanic data provided for assignment contains a Field called Passenger Name,

    that name contains comma in between but surrounded by quotes, with that i able to identify this full name

    but while loading into table, creates improper data because of that comma

    Kindly help
     
    #8
  9. 1a9e3f5

    1a9e3f5 Member
    Alumni

    Joined:
    Feb 7, 2019
    Messages:
    7
    Likes Received:
    0
    Hi Rakesh,
    I am not able to update sql table with titanic data. Please see attached notepad. Can you please help?

    Regards
     
    #9
  10. 1a9e3f5

    1a9e3f5 Member
    Alumni

    Joined:
    Feb 7, 2019
    Messages:
    7
    Likes Received:
    0
    Hi Rakesh,
    I am not able to update sql table with titanic data. Please see attached notepad. Can you please help? forgot to attach the file before.

    Regards
     

    Attached Files:

    #10
  11. Rakesh_236

    Rakesh_236 Active Member

    Joined:
    Dec 27, 2018
    Messages:
    22
    Likes Received:
    2
    Hi,
    You need to create the table in hive not in MYSQL. All the load commands are for Hive.

    regards
    Rakesh
     
    #11
  12. Rakesh_236

    Rakesh_236 Active Member

    Joined:
    Dec 27, 2018
    Messages:
    22
    Likes Received:
    2
    Hi Boopathy,

    That is correct. It was intentional and expectation is to use hive built in string function to remove the additional comma from passenger name field.
    Functions that will help you remove the additional comma: regexp_replace and regexp_extract.

    I will share the solution this weekend.

    Regards
    Rakesh Srivastva
     
    #12
  13. 1a9e3f5

    1a9e3f5 Member
    Alumni

    Joined:
    Feb 7, 2019
    Messages:
    7
    Likes Received:
    0
    Is there any assignment for this week on spark ?
     
    #13
  14. 1a9e3f5

    1a9e3f5 Member
    Alumni

    Joined:
    Feb 7, 2019
    Messages:
    7
    Likes Received:
    0
    Is there a difference in CCA159 and CCA175 in the way job market is looking ?
    level of effort , time, market value ? any pros and cons on either one ?
     
    #14
  15. Boopathy_1

    Boopathy_1 Member

    Joined:
    Feb 8, 2019
    Messages:
    4
    Likes Received:
    0
    #15
  16. Kunal_114

    Kunal_114 Member

    Joined:
    Mar 27, 2018
    Messages:
    7
    Likes Received:
    0
    #16
  17. _62919

    _62919 Member

    Joined:
    Jun 1, 2019
    Messages:
    12
    Likes Received:
    0
    hello!
    i am trying to import table through sqoop using the following command in terminal in virtual box.
    sqoop import --connect jdbc:mysql://localhost/training --username training --password training --table countries;

    I am getting this error:
    ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
    ear 'sqoop import --connect jdbc:mysql://localhost/training --username training --pas' at line 1

    please help me out.
     
    #17

Share This Page