namenode and datanode incompatible cluster ids

Discussion in 'Big Data and Analytics' started by DIVAKAR PISUPATI, May 11, 2016.

  1. DIVAKAR PISUPATI

    Alumni

    Joined:
    Apr 23, 2016
    Messages:
    11
    Likes Received:
    0
    Hi
    I have some issue with datanode not starting properly, so i checked logs of datanode and found that namenode and datanode are not in sync, below is the error message from logs: Both of the nodes are in different cluster ids. I tried removing the datanode cluster from name node and restart the service, but no luck..your help in this regard is highly appreciated..thanks..Divakar
    Error:
    2016-05-11 20:37:26,720 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
    2016-05-11 20:37:26,801 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 43552
    2016-05-11 20:37:26,802 INFO org.mortbay.log: jetty-6.1.26
    2016-05-11 20:37:27,727 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43552
    2016-05-11 20:37:28,528 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:50075
    2016-05-11 20:37:28,824 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = divakar
    2016-05-11 20:37:28,824 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
    2016-05-11 20:37:29,161 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
    2016-05-11 20:37:29,305 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2016-05-11 20:37:29,550 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
    2016-05-11 20:37:29,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
    2016-05-11 20:37:29,781 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
    2016-05-11 20:37:29,850 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:54310 starting to offer service
    2016-05-11 20:37:29,917 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2016-05-11 20:37:29,931 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
    2016-05-11 20:37:31,221 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /usr/local/hadoop/hadoop_data/hdfs/datanode/in_use.lock acquired by nodename 4048@divakarhost
    2016-05-11 20:37:31,228 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop/hadoop_data/hdfs/datanode: namenode clusterID = CID-4dab02fe-69c2-4a0b-85d9-294fe1af2204; datanode clusterID = CID-0a7d2eda-2f36-49ef-9919-5c090c94aa18
    2016-05-11 20:37:31,232 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:54310. Exiting.
    java.io.IOException: All specified directories are failed to load.
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
     
    #1
  2. Karthik Shivana

    Karthik Shivana Moderator
    Simplilearn Support Alumni

    Joined:
    Apr 1, 2016
    Messages:
    688
    Likes Received:
    31
    Hi Divakar,

    Please check namenode version IP and Datanode IP version are same, if not. Please make it same.

    Regards
    Karthik
     
    #2
  3. DIVAKAR PISUPATI

    Alumni

    Joined:
    Apr 23, 2016
    Messages:
    11
    Likes Received:
    0
    Hi Karthik
    Can you pls elaborate?
    I checked name node and data node(as below), however I don't see any option to change the ip address.. pls help
    #Thu May 12 19:30:25 IST 2016
    namespaceID=1953714539
    clusterID=CID-751dad6e-06a0-4e99-9320-1607202ddd23
    cTime=0
    storageType=NAME_NODE
    blockpoolID=BP-104761465-127.0.1.1-1463061625662
    layoutVersion=-63

    #Mon May 02 15:12:31 IST 2016
    storageID=DS-a8e82d60-d2b4-43d6-ab0b-d6433bf2d7d7
    clusterID=CID-0a7d2eda-2f36-49ef-9919-5c090c94aa18
    cTime=0
    datanodeUuid=701ae9d6-a79f-4eb0-8337-d3542299ec0f
    storageType=DATA_NODE
    layoutVersion=-63
     
    #3
  4. Karthik Shivana

    Karthik Shivana Moderator
    Simplilearn Support Alumni

    Joined:
    Apr 1, 2016
    Messages:
    688
    Likes Received:
    31
    Hi Divakar,

    Please share your contact number, so that we will try to connect, otherwise call to simplilearn customer care.

    Apologize, Its not IP, its Version ID of NN and DN.

    Regards
    Karthik
     
    #4

Share This Page