Welcome to the Simplilearn Community

Want to join the rest of our members? Sign up right away!

Sign Up

Big Data Hadoop and Spark Developer ( Oct 24th - Dec 12th,2020) | Amit Singh

Hi,
I dont have permission to do anything other than write in my web console. This msg im gettingas output
"mkdir: Permission denied: user=****, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x"
where should I complain??
 
Hi,
I dont have permission to do anything other than write in my web console. This msg im gettingas output
"mkdir: Permission denied: user=****, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x"
where should I complain??
Hi,

Thank you for reaching to us.
As requested, I have shared below screenshot for your further reference:
upload_2020-10-30_0-29-21.png

I hope this will help you out.

Regards
Koyel
 

Bhagyalaxmi

Active Member
Hello sir, i have create table in mysql now i want to know is there any other way to load data into sql table or i have to manually enter all the rows in the table?

as in hive we are creating table later we are loading data of files into that tables as load inpath.......etc
such a way we load data of csv file into sql table.
 
Last edited:

Koyel Sinha Chowdhury

Well-Known Member
Staff member
Problem Statement: PV Consulting is one of the top consulting firms for big data projects. They mostly help big and small companies to analyze their data.
For Spark and Hadoop MR application, they started using YARN as a resource manager. Your task is to provide the following information for any job which is submitted to YARN using YARN console and YARN Cluster UI:

  1. Who has submitted the job? :- Client right?
  2. To which YARN queue is a job submitted? :- there is 3 queue can you tell me which one is it
  3. How much time did it take to finish the job? :- ?
  4. List of all running jobs on YARN
  5. How to kill a job using YARN?
  6. Check logs using YARN
please can you help me with this project
this all are the practical questions and simple question answer?
Hi Learner,
Please use the URL,
For Resource Manager: http://slbdh.corestack.io:50014
For HistoryServer : http://slbdh.corestack.io:50013

Kindly refer the below screenshots for your further reference:
 

Attachments

  • Capture_course resources.PNG
    Capture_course resources.PNG
    683.9 KB · Views: 22
  • Capture_History.PNG
    Capture_History.PNG
    123.5 KB · Views: 21
  • Capture_Yarn.PNG
    Capture_Yarn.PNG
    145.3 KB · Views: 19
  • Capture_Resource Manager.PNG
    Capture_Resource Manager.PNG
    104.3 KB · Views: 18

Bhagyalaxmi

Active Member
ip-10-0-42-218 login: patelbhagya1709gmailPassword: Last login: Tue Dec 1 17:59:50 on pts/3856[patelbhagya1709gmail@ip-10-0-42-218 ~]$ mysql -h sqoopdb.slbdh.cloudlabs.com -u patelbhagya1709gmail -pEnter password: Welcome to the MariaDB monitor. Commands end with ; or \g.Your MySQL connection id is 19713Server version: 8.0.20 MySQL Community Server - GPLCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> use patelbhagya1709gmail;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedMySQL [patelbhagya1709gmail]> LOAD DATA INFILE '/user/patelbhagya1709gmail/projects/Stockcompanies.csv' INTO TABLE Stockcompanies FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS;ERROR 1045 (28000): Access denied for user 'patelbhagya1709gmail'@'%' (using password: YES)

can you tell me why access is denied
why i am not able to load csv gfile into my sql
 

Bhagyalaxmi

Active Member
Create a data pipeline using sqoop to pull the data from the table below from MYSQL server into Hive.

can you tell me how to load csv file into my sql server, exporting part i know mysql to hive .........

we have exported emp table into hive through sqoop that was manually created and manually inserted data so over here is there any way to load csv file into my sql table to get csv data

i tried but access is denied to my user
 

Bhagyalaxmi

Active Member
hello sir, in 2nd project there is no separate column for country then how we can partition by country can you please tell me?

also how to split column? please answer me
 
ip-10-0-42-218 login: patelbhagya1709gmailPassword: Last login: Tue Dec 1 17:59:50 on pts/3856[patelbhagya1709gmail@ip-10-0-42-218 ~]$ mysql -h sqoopdb.slbdh.cloudlabs.com -u patelbhagya1709gmail -pEnter password: Welcome to the MariaDB monitor. Commands end with ; or \g.Your MySQL connection id is 19713Server version: 8.0.20 MySQL Community Server - GPLCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> use patelbhagya1709gmail;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedMySQL [patelbhagya1709gmail]> LOAD DATA INFILE '/user/patelbhagya1709gmail/projects/Stockcompanies.csv' INTO TABLE Stockcompanies FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS;ERROR 1045 (28000): Access denied for user 'patelbhagya1709gmail'@'%' (using password: YES)

can you tell me why access is denied
why i am not able to load csv gfile into my sql
Having same access denied issue. Unable to load CSV into MySQL table.
Without loading we can't fullfill the requested task.
 
Last edited:

Bhagyalaxmi

Active Member
scala> myfirstrdd1.saveAsTextFile("D:\\bhagya1")
20/12/06 17:16:45 ERROR Executor: Exception in task 0.0 in stage 6.0 (TID 10)
java.io.IOException: (null) entry in command string: null chmod 0644 D:\bhagya1\_temporary\0\_temporary\attempt_20201206171645_0012_m_000000_0\part-00000

it create folder but havn't saved anything in that folder
what is error can you tell me?
 

Bhagyalaxmi

Active Member
scala> map
map map_concat map_entries map_filter map_from_arrays map_from_entries map_keys map_values map_zip_with

scala> val rdd2 = sc.textFile("D:\\apple-sales.csv")
rdd2: org.apache.spark.rdd.RDD[String] = D:\apple-sales.csv MapPartitionsRDD[6] at textFile at <console>:24

scala> val mappping = rdd2.map(x => (x(7),1))
mappping: org.apache.spark.rdd.RDD[(Char, Int)] = MapPartitionsRDD[7] at map at <console>:25

scala> val mappping = rdd2.map(x => x.split(" , "))
mappping: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[8] at map at <console>:25

scala> val countryName = mappping.map(x => (x(7),1))
countryName: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[9] at map at <console>:25

scala> val result = countryName.reduceByKey((a,b) => (a+b))
result: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[10] at reduceByKey at <console>:25

scala> result.collect
20/12/06 22:51:55 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 3)
java.lang.ArrayIndexOutOfBoundsException: 7
at $line26.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.$anonfun$countryName$1(<console>:25)
at $line26.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$Lambda$2756/652265534.apply(Unknown Source)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
 
Last edited:
scala> val rdd2 = sc.textFile("D:\\apple-sales.csv")
rdd2: org.apache.spark.rdd.RDD[String] = D:\apple-sales.csv MapPartitionsRDD[6] at textFile at <console>:24

scala> val mappping = rdd2.map(x => (x(7),1))
mappping: org.apache.spark.rdd.RDD[(Char, Int)] = MapPartitionsRDD[7] at map at <console>:25

scala> val mappping = rdd2.map(x => x.split(" , "))
mappping: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[8] at map at <console>:25

scala> val result = countryName.reduceByKey((a,b) => (a+b))
result: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[10] at reduceByKey at <console>:25

This will work ...you are trying to do map multiple times.
 
scala> myfirstrdd1.saveAsTextFile("D:\\bhagya1")
20/12/06 17:16:45 ERROR Executor: Exception in task 0.0 in stage 6.0 (TID 10)
java.io.IOException: (null) entry in command string: null chmod 0644 D:\bhagya1\_temporary\0\_temporary\attempt_20201206171645_0012_m_000000_0\part-00000

it create folder but havn't saved anything in that folder
what is error can you tell me?

Show me this in class.
 
Top