Welcome to the Simplilearn Community

Want to join the rest of our members? Sign up right away!

Sign Up

Big data on AWS | Pratik Gosawi | Apr 10 -May 15

Hi Students,
I've uploaded a PDF in drive and here in forum as well, defining a simple explanation of IoT services on AWS that will be helpful to you guys for your exam.

Please check it and I will be uploading more such docs.
 

Attachments

  • IoT - Internet of Things on AWS - Simple Explanation.pdf
    44.3 KB · Views: 1
Hi Students,
Hope you guys doing well.

I'm uploading another document that contains the main difference points between Kinesis Data Stream and SQS.
This will help you a lot from an exam point of view.

Happy Learning!
 

Attachments

  • KDS vs SQS.pdf
    26 KB · Views: 1

Akshay Kumar_10

Active Member
Hi Team,

I am using the Kinesis data stream and connected to "SOURCE_SQL_STREAM_001" trying to discover the schema but getting the issue .

Region : N-Virginia & Ohio in both tried

Generating the data from the "amazon-kinesis-data-generator" and its working fine which i saw in Data Streams Metrics that Incoming Data -sum count .



1619348887814.png


1619348871202.png
 
Hi Team,

I am using the Kinesis data stream and connected to "SOURCE_SQL_STREAM_001" trying to discover the schema but getting the issue .

Region : N-Virginia & Ohio in both tried

Generating the data from the "amazon-kinesis-data-generator" and its working fine which i saw in Data Streams Metrics that Incoming Data -sum count .



1619348887814.png


1619348871202.png
Hi Akshay,
Let me know are you trying to run SQL queries using Kinesis Data Analytics?
 

Akshay Kumar_10

Active Member
Hi Akshay,
Let me know are you trying to run SQL queries using Kinesis Data Analytics?
Hi Prateik,Thanks for your reply . Yes i am using Kinesis Data Analytics to run the SQL queries which is receiving to data stream and Firehouse ,yes my source is data stream and then using data analytics to run the SQL queries .
 

Akshay Kumar_10

Active Member
Hi Prateik,Thanks for your reply . Yes i am using Kinesis Data Analytics to run the SQL queries which is receiving to data stream and Firehouse ,yes my source is data stream and then using data analytics to run the SQL queries .
And also i am sure that what are the data i am generating it correct and receiving to data stream but while running same in data analytics using SQL throwing any error so can able to push into the ES .
 
Hi Prateik,Thanks for your reply . Yes i am using Kinesis Data Analytics to run the SQL queries which is receiving to data stream and Firehouse ,yes my source is data stream and then using data analytics to run the SQL queries .
Hi Akshay,
There seems to be an error in your queries.
I'll suggest you to look in to it or else send me the code, I'll try to fix it
 

Akshay Kumar_10

Active Member
Hi Akshay,
There seems to be an error in your queries.
I'll suggest you to look in to it or else send me the code, I'll try to fix it
Hi Prateik,
Thanks for replying !

Scenario :
While configuring 1st the Data Analytics below queries is running after "Click and Save " the data analytic and again going to SQL to fetch the same with below queries its throwing an error might be issue with any type of access which need to be given on Firehouse /Data Anayltics ,i tried lot of tests but still same issue so if you got this issue in real-time if yo can suggest.

Please reveiw my Queries :

CREATE OR REPLACE STREAM "TEMP_STREAM" (
"iotName" varchar (40),
"iotValue" integer,
"ANOMALY_SCORE" DOUBLE);
-- Creates an output stream and defines a schema
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
"iotName" varchar(40),
"iotValue" integer,
"ANOMALY_SCORE" DOUBLE,
"created" TimeStamp);

-- Compute an anomaly score for each record in the source stream
-- using Random Cut Forest
CREATE OR REPLACE PUMP "STREAM_PUMP_1" AS INSERT INTO "TEMP_STREAM"
SELECT STREAM "iotName", "iotValue", ANOMALY_SCORE FROM
TABLE(RANDOM_CUT_FOREST(
CURSOR(SELECT STREAM * FROM "SOURCE_SQL_STREAM_001")
)
);

-- Sort records by descending anomaly score, insert into output stream
CREATE OR REPLACE PUMP "OUTPUT_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM "iotName", "iotValue", ANOMALY_SCORE, ROWTIME FROM "TEMP_STREAM"
ORDER BY FLOOR("TEMP_STREAM".ROWTIME TO SECOND), ANOMALY_SCORE DESC;
 
Last edited:
Hi Prateik,
Thanks for replying !

Scenario :
While configuring 1st the Data Analytics below queries is running after "Click and Save " the data analytic and again going to SQL to fetch the same with below queries its throwing an error might be issue with any type of access which need to be given on Firehouse /Data Anayltics ,i tried lot of tests but still same issue so if you got this issue in real-time if yo can suggest.

Please reveiw my Queries :

CREATE OR REPLACE STREAM "TEMP_STREAM" (
"iotName" varchar (40),
"iotValue" integer,
"ANOMALY_SCORE" DOUBLE);
-- Creates an output stream and defines a schema
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
"iotName" varchar(40),
"iotValue" integer,
"ANOMALY_SCORE" DOUBLE,
"created" TimeStamp);

-- Compute an anomaly score for each record in the source stream
-- using Random Cut Forest
CREATE OR REPLACE PUMP "STREAM_PUMP_1" AS INSERT INTO "TEMP_STREAM"
SELECT STREAM "iotName", "iotValue", ANOMALY_SCORE FROM
TABLE(RANDOM_CUT_FOREST(
CURSOR(SELECT STREAM * FROM "SOURCE_SQL_STREAM_001")
)
);

-- Sort records by descending anomaly score, insert into output stream
CREATE OR REPLACE PUMP "OUTPUT_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM "iotName", "iotValue", ANOMALY_SCORE, ROWTIME FROM "TEMP_STREAM"
ORDER BY FLOOR("TEMP_STREAM".ROWTIME TO SECOND), ANOMALY_SCORE DESC;
Hey Akshay,
Apologize for the late reply, I'm busy due to some personal emergency
I analyzed your code it seems fine to me.
Check whether you have assigned a proper IAM role or else we will have to share screen .
This is one of the practice examples from the documentation, when I ran it some time ago, it was running fine.
It could be like you have missed some step.
 

Akshay Kumar_10

Active Member
Hey Akshay,
Apologize for the late reply, I'm busy due to some personal emergency
I analyzed your code it seems fine to me.
Check whether you have assigned a proper IAM role or else we will have to share screen .
This is one of the practice examples from the documentation, when I ran it some time ago, it was running fine.
It could be like you have missed some step.
Hi Prateik,

Thanks for your reply and Please take care !

I have tried multiple IAM role which connect/give the access to Elastic Search Domain Service and also to firehouse.
Might be screen sharing in the correct option if you have a look ,might be this Saturday/Sunday during our classes one you got a chance will share my screen quickly or i am ready anytime whenever you are free.

I will be ready with the setup so you can have a look quickly.

Thanks Again for replying :) Stay Safe ! Stay Healthy !
 

alma.ibanez

Customer
Customer
Hello Team;

I try to create a domane in Elastic Serch and i have the next issue:

1621218895571.png

And wheni create an Instance in EC2 I hav no permission:

1621218982242.png

when i try to conect with putty, i can`t :

1621219052112.png

Could you help me, please?
 
Top