what is the difference between local mode and mapreduce mode

Discussion in 'Big Data and Analytics' started by Parminder Sohal(2946), Oct 18, 2015.

  1. Parminder Sohal(2946)

    Parminder Sohal(2946) Active Member
    Trainer

    Joined:
    Aug 25, 2014
    Messages:
    39
    Likes Received:
    6
    Local mode:
    All scripts are run on a single machine without requiring Hadoop MapReduce and HDFS. This can be useful for developing and testing Pig logic. If you’re using a small set of data to developer or test your code, then local mode could be faster than going through the MapReduce infrastructure.
    Local mode doesn’t require Hadoop. When you run in Local mode, the Pig program runs in the context of a local Java Virtual Machine, and data access is via the local file system of a single machine. Local mode is actually a local simulation of MapReduce in Hadoop’s LocalJobRunner class.
    MapReduce mode (also known as Hadoop mode): Pig is executed on the Hadoop cluster. In this case, the Pig Script gets converted into a series of MapReduce jobs that are then run on the Hadoop cluster.

    [​IMG]

    If you have a terabyte of data that you want to perform operations on and you want to interactively develop a program, you may soon find things slowing down considerably, and you may start growing your storage. Local mode allows you to work with a subset of your data in a more interactive manner so that you can figure out the logic (and work out the bugs) of your Pig program.

    After you have things set up as you want them and your operations are running smoothly, you can then run the script against the full data set using MapReduce mode.
     
    #1

Share This Page