Spark and Hbase integration program using Eclipse

Discussion in 'Big Data and Analytics' started by _55362, Feb 12, 2019.

  1. _55362

    _55362 Member

    Joined:
    Jan 10, 2019
    Messages:
    3
    Likes Received:
    0
    Dear All,

    I have tried to do CRUD operations in HBase and Spark (Scala) using Eclipse IDE.

    Below steps, I have tried. (Eclipse IDE)

    Step 1: Loaded Spark Jar Files (2.4 version)
    Step 2: Loaded HBase Jar Files
    Step 3: I have written below code

    package com.zaloni.mgohain.sparkHbaseIntegration.services
    import org.apache.hadoop.hbase.client.{HBaseAdmin,HTable,Put}import org.apache.hadoop.hbase.mapreduce.TableOutputFormatimport org.apache.hadoop.hbase.util.Bytesimport org.apache.hadoop.hbase.{HBaseConfiguration,HColumnDescriptor,HTableDescriptor}
    objectEmployee{def main(args:Array[String]){if(args.length !=1){System.err.println("In correct number of arguments "+ args.length)System.out.println("Please provide correct arguments.")System.exit(1)}
    val hbaseConf =HBaseConfiguration.create()
    val tableName ="employee"
    hbaseConf.set(TableOutputFormat.OUTPUT_TABLE, tableName)
    hbaseConf.set("hbase.zookeeper.quorum","quickstart.cloudera")
    hbaseConf.set("hbase.zookeeper.property.client.port","2181")
    val admin =newHBaseAdmin(hbaseConf)
    val cfProfessionalData =Bytes.toBytes("professional_data")
    val cfPersonalData =Bytes.toBytes("personal_data")if(!admin.isTableAvailable(tableName)){
    val tableDesc =newHTableDescriptor(tableName)
    tableDesc.addFamily(newHColumnDescriptor(cfProfessionalData))
    tableDesc.addFamily(newHColumnDescriptor(cfPersonalData))}
    val hTable =newHTable(hbaseConf,tableName)//val records = sc.textFile(args(0))
    val put =newPut(Bytes.toBytes("e_1"))
    val eId =Bytes.toBytes("Emp_id")
    val name =Bytes.toBytes("Name")
    val dsgtn =Bytes.toBytes("Designation")
    val doj =Bytes.toBytes("DOJ")
    val addr =Bytes.toBytes("Address")
    val phn =Bytes.toBytes("Phone")
    val dob =Bytes.toBytes("DOB")
    put.add(cfProfessionalData, eId,Bytes.toBytes(1))
    put.add(cfProfessionalData, name,Bytes.toBytes("Mridul Gohain"))
    put.add(cfProfessionalData, dsgtn,Bytes.toBytes("SE"))
    put.add(cfProfessionalData, doj,Bytes.toBytes("15-07-2015"))
    put.add(cfPersonalData, addr,Bytes.toBytes("Chabua"))
    put.add(cfPersonalData, phn,Bytes.toBytes("9859559606"))
    put.add(cfPersonalData, dob,Bytes.toBytes("04-10-1991"))
    hTable.put(put)
    hTable.close()}}
     
    #1
  2. ruhi.jain

    ruhi.jain Well-Known Member
    Simplilearn Support

    Joined:
    Jun 7, 2018
    Messages:
    226
    Likes Received:
    5

    Please let me know in case if you are facing any issue in this code.
     
    #2

Share This Page