Is shared block in HDFS possible?

Discussion in 'Big Data and Analytics' started by Sandeep_250, Oct 10, 2017.

  1. Sandeep_250

    Sandeep_250 Active Member
    Alumni

    Joined:
    Jun 21, 2016
    Messages:
    20
    Likes Received:
    1
    Let’s say I have a file of 130 MB and I put it on HDFS. The first block will have 128 MB and 2nd block will have 2 MB. Now if there is another file of 132 MB also put in HDFS then again, this one will have first block of 128 MB and 2nd block of 4 MB. Can the 2nd blocks of each of these files be shared in single block? Means a block containing 2 MB of one file and 4 MB of another file. Is it possible?
     
    #1
  2. _6230

    _6230 Well-Known Member
    Alumni

    Joined:
    Apr 4, 2017
    Messages:
    152
    Likes Received:
    8
    If file A of 132 MB is stored in HDFS with 2 blocks. These blocks only referred to file A and never available for any other files to share. The content is available in memory until datanode and name node is not formatted.
     
    #2
  3. Sandeep_250

    Sandeep_250 Active Member
    Alumni

    Joined:
    Jun 21, 2016
    Messages:
    20
    Likes Received:
    1
    One of the trainer told it can be shared but I doubted. I will still like to have clarification from SimpliLearn support on this. They seem to keep mum until now.
     
    #3
  4. Sandeep_250

    Sandeep_250 Active Member
    Alumni

    Joined:
    Jun 21, 2016
    Messages:
    20
    Likes Received:
    1
    Its really surprising that until now none of the SimpliLearn support member noticed this thread.
     
    #4
  5. Ravikumar Girani

    Ravikumar Girani Active Member
    Alumni

    Joined:
    Jun 8, 2017
    Messages:
    23
    Likes Received:
    0
    Hi Dude,

    As per my knowledge you will be a client having a files of 130 MB and 132 MB file.HDFS will work on your 1st file like below
    ->split the file into the blocks:130/128=1.01.So will have two block ,fist block of size 128 MB and last of the size 2MB.
    splitting is done by name node not by client.
    ->Now you have to place the file into the datanodes but whether the client(you) knows which all DataNodes are having free space?
    No. So Client will contact NN and will ask-Hey NN,I have a 130 MB of files which I want to keep on DataNodes. Please tell me the DataNodes where I should keep it.
    ->Then Name node take the metadat from client file,name,size,block and Name node will resopnse to client with available data nodes like 1,3,5.
    ->After cliend will place the files on those data nodes.

    If you want to keep another file of 132 MB you have contact to Name node .....visualize and imagine answer yourself...I hope you got it.

    But we have to keep in mind HDFS follows WORM principle.
     
    #5
  6. Sandeep_250

    Sandeep_250 Active Member
    Alumni

    Joined:
    Jun 21, 2016
    Messages:
    20
    Likes Received:
    1
    I know all these. I am expecting answer from the SimpliLearn support which they are not responding.
     
    #6

Share This Page