Big Data Hadoop and Spark Developers | Gautam

Discussion in 'Big Data and Analytics' started by Koyel Sinha Chowdhury, Jul 23, 2019.

  1. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    Will share the Cassandra installation doc by tomorrow.
     
    #51
  2. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    Delete all the files from HDFS. $hadoop namenode -format. Also delete all files under /tmp folder. #cd /tmp #rm -rf /tmp/*
     
    #52
  3. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    Try to run Zookeepre. #service zookeeper-server start. Check status service zookeeper-server status. If Zookeeper is running fine, ignore the issue.
     
    #53
  4. Shailendra Parauha

    Shailendra Parauha Active Member

    Joined:
    Jul 26, 2019
    Messages:
    21
    Likes Received:
    0
    thanks able to run the zookeeper ignoring this error
     
    #54
  5. Shailendra Parauha

    Shailendra Parauha Active Member

    Joined:
    Jul 26, 2019
    Messages:
    21
    Likes Received:
    0
    Hi Gautam,
    In eclipse, Inside sample folder I can't see src/main/scala folder, although I followd all the steps given by you.
    Is there any thing we have to do with classpath for sbt?
    Please see the attached screen shot "Eclipse_scala_error.png".

    'sbt package' and 'sbt eclispe' executed successfully, below is details

    C:\myhdfsprj\spark>sbt package
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
    [info] Updated file C:\myhdfsprj\spark\project\build.properties: set sbt.version to 1.2.8
    [info] Loading settings for project global-plugins from plugins.sbt ...
    [info] Loading global plugins from C:\Users\sparauha\.sbt\1.0\plugins
    [info] Loading project definition from C:\myhdfsprj\spark\project
    [info] Updating ProjectRef(uri("file:/C:/myhdfsprj/spark/project/"), "spark-build")...
    [info] Done updating.
    [info] Loading settings for project spark from build.sbt ...
    [info] Set current project to sample (in build file:/C:/myhdfsprj/spark/)
    [info] Updating ...
    [info] Done updating.
    [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
    [info] Compiling 1 Scala source to C:\myhdfsprj\spark\target\scala-2.11\classes ...
    [info] Done compiling.
    [info] Packaging C:\myhdfsprj\spark\target\scala-2.11\sample_2.11-0.1.0-SNAPSHOT.jar ...
    [info] Done packaging.
    [success] Total time: 5 s, completed Sep 4, 2019 7:40:58 PM


    C:\myhdfsprj\spark>sbt eclipse
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
    [info] Loading settings for project global-plugins from plugins.sbt ...
    [info] Loading global plugins from C:\Users\sparauha\.sbt\1.0\plugins
    [info] Loading project definition from C:\myhdfsprj\spark\project
    [info] Loading settings for project spark from build.sbt ...
    [info] Set current project to sample (in build file:/C:/myhdfsprj/spark/)
    [info] About to create Eclipse project files for your project(s).
    [info] Successfully created Eclipse project files for project(s):
    [info] sample
    C:\myhdfsprj\spark>
    Thanks,
    Shailendra
     

    Attached Files:

    #55
  6. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    In the left-hand side navigator, I can see src/main/package and the WordCOunt.scala inside it in Eclipse. Are you referring to some other file?
     
    #56
  7. Shailendra Parauha

    Shailendra Parauha Active Member

    Joined:
    Jul 26, 2019
    Messages:
    21
    Likes Received:
    0
    Hi Gautam,

    I am using google cloud only one node as master and worker both.
    I am using apache spark 2.4.3 over cdh5

    Below command is throwing error:
    bash-4.2$ ./spark-submit --class WordCount --master yarn --deploy-mode client /usr/spark-example/target/sample_2.11
    -0.1.0-SNAPSHOT.jar
    Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR
    or YARN_CONF_DIR must be set in the environment.

    I copied files hdfs-site.xml and yarn-site.xml to the dir /usr/spark-2.4.3-bin-hadoop2.7/conf but it doens' not work
    Now i have created a file spark-env.sh from spark-env.sh.template but I don't know what all changes needed in the spark-env.sh file to run the master as yarn command.

    Please let me know how can I fix this issue...

    Thanks,
    Shailendra
     
    #57
  8. Koyel Sinha Chowdhury

    Koyel Sinha Chowdhury Well-Known Member

    Joined:
    Feb 14, 2019
    Messages:
    54
    Likes Received:
    5
    Hi Shailendra,

    I have shared the commands given below :

    #While running spark using Yarn, you need to add following line in to spark-env.sh

    >>export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

    #Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.

    I hope this will help you out.
     
    #58
    Shailendra Parauha likes this.
  9. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    Hi Shailendra,

    export HADOOP_CONF_DIR=/etc/hadoop/conf

    try the command again.
     
    #59
  10. Caelyn Leo

    Caelyn Leo Member

    Joined:
    May 29, 2018
    Messages:
    2
    Likes Received:
    0
    Hi,
    Unable to connect to hive-server2. google cloud services

    beeline -u jdbc:hive2://localhost:10001/default

    scan complete in 2msConnecting to jdbc:hive2://localhost:10001/defaultCould not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10001/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)Beeline version 1.1.0-cdh5.16.2 by Apache Hive
     
    #60
  11. Shailendra Parauha

    Shailendra Parauha Active Member

    Joined:
    Jul 26, 2019
    Messages:
    21
    Likes Received:
    0
    Yes it work thanks for your help :)
     
    #61
  12. Gautam Pal

    Gautam Pal Customer
    Customer

    Joined:
    Jul 23, 2019
    Messages:
    27
    Likes Received:
    1
    Install Hive Server 2
    -------------------------
    sudo yum install hive-server2 -y
    Start Hive Server 2
    ---------------------------
    sudo service hive-server2 start
    Check status of Hive server 2
    ------------------------------
    sudo service hive-server2 status
    Add the following in /etc/hive/conf/hive-site.xml file:
    ----------------------------------------------------------
    <property>
    <name>hive.server2.thrift.port</name>
    <value>10001</value>
    <description>TCP port number to listen on, default 10000</description>
    </property>
    <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>localhost</value>
    <description>TCP port number to listen on, default 10000</description>
    </property>
    Connect
    -----------
    beeline -u jdbc:hive2://localhost:10001/default
     
    #62
  13. Caelyn Leo

    Caelyn Leo Member

    Joined:
    May 29, 2018
    Messages:
    2
    Likes Received:
    0
    I have already installed and set up the hive-site.xml file.
    The error msg previously posted was after connecting to beeline via the command.

    Tried again and further checked with show tables gives : No current connection


     
    #63
  14. Koyel Sinha Chowdhury

    Koyel Sinha Chowdhury Well-Known Member

    Joined:
    Feb 14, 2019
    Messages:
    54
    Likes Received:
    5
    # Connect command:
    !connect jdbc:hive2://<hiveserver2-hostname-or-ip>:10000/<database-name>
    !connect jdbc:hive2://localhost:10000 org.apache.hive.jdbc.HiveDriver

    Also I have shared the below link for your reference:
    https://stackoverflow.com/questions/28898936/beeline-not-able-to-connect-to-hiveserver2
     
    #64

Share This Page