This is documentation for MapR Version 5.0. You can also refer to MapR documentation for the latest release.

Skip to end of metadata
Go to start of metadata

The method that you use to configure the MapReduce mode is based on how the ecosystem client submits jobs to the cluster:

Component BehaviorMethod
The ecosystem client directly submits jobs to the cluster.Configure the MapReduce Mode for Ecosystem Clients.
The ecosystem client has a service that submits jobs to the cluster.Configure the MapReduce Mode for Ecosystem Service.

Configuring MapReduce Mode for Ecosystem Clients

For Ecosystem clients that directly submit MapReduce jobs to the cluster, the MapReduce mode will default to the MapReduce mode of the client node from which the job is submitted.

In the following figure, Pig is installed on a client node with default_mode=yarn. Pig directly submits the application to the ResourceManager in the cluster.

When the ecosystem component connects directly to the cluster (not through a service), you can set the MapReduce version for the job or application in an environment variable. The mode that you set in an environment variable overrides the default MapReduce mode that is set on the client node and the cluster.

To set the MapReduce mode for ecosystem component that connect directly to the cluster:

  1. Open a terminal on the client node.
  2. Enter one of the following commands on the shell:
    • export MAPR_MAPREDUCE_MODE= yarn
    • export MAPR_MAPREDUCE_MODE= classic
  3. Launch the ecosystem client and submit the job or application.

Configuring MapReduce Mode for Ecosystem Service

Some ecosystem clients communicate with the cluster through a service. In this case, the MapReduce mode that is configured for the ecosystem service has a higher priority and cannot be overridden by the client.

By default, the ecosystem service will run MapReduce jobs and applications using the MapReduce mode of the cluster. However, you can set the MapReduce mode for the following ecosystem services:

  • Hue
  • Httpsfs
  • Hiveserver2
  • Oozie

In the following figure, the Hive client is installed on a client node with a default_mode=yarn and the Hive service is installed on a cluster with default_mode=yarn. However, since the Hive server has a MAPR_MAPREDUCE_MODE=classic, the job is submitted to the JobTracker.

To configure the MapReduce Mode for an ecosystem service:

  1. Open the warden.<servicename>.conf file for the particular ecosystem service. The warden.<servicename>.conf files are located in the following directory: /opt/mapr/conf/conf.d
  2. Edit the MAPR_MAPREDUCE_MODE in the service.env variable.
    The following values are valid for the MAPR_MAPREDUCE_MODE:

    service.env ValueDescription

    MAPR_MAPREDUCE_MODE=classic

    Specifies that the service submits MapReduce v1 jobs to the cluster. Do not include quotes around this value.

    MAPR_MAPREDUCE_MODE=yarn

    Specifies that the service submits MapReduce v2 applications to the cluster. Do not include quotes around this value.
    "MAPR_MAPREDUCE_MODE=default"

    Specifies that the service should submit MapReduce jobs/applications based on the cluster’s default MapReduce mode.

  3. Restart the ecosystem service.

 

  • No labels