Skip to end of metadata
Go to start of metadata

The file /opt/mapr/hadoop/hadoop-<version>/conf/mapred-site.xml contains configuration information that overrides the default values for MapReduce parameters. Overrides of the default values for core configuration properties are stored in the core-site.xml file.

To override a default value, specify the new value within the <configuration> tags, using the following format:

The following configuration tables describes the possible entries to place in the <name> and <value> tags. The <description> tag is optional but recommended for maintainability. There are three parts to mapred-site.xml:

Icon

You can examine the current configuration information for this node by using the hadoop conf -dump command from a command line.

Default core-site.xml file

JobTracker Configuration

Changing any parameters in this section requires a JobTracker restart.

Parameter

Value

Description

mapreduce.job.counters.counter.name.max64

Limit on the length of counter names in jobs. Names exceeding this limit will be truncated.

mapreduce.job.counters.group.name.max128

Limit on the length of counter group names in jobs. Names exceeding this limit will be truncated.

mapred.job.tracker

maprfs:///

JobTracker address ip:port or use uri maprfs:/// for default cluster or maprfs:///mapr/san_jose_cluster1 to connect 'san_jose_cluster1' cluster.
Replace localhost by one or more ip addresses for JobTracker.

mapred.jobtracker.port

9001

Port on which JobTracker listens. Read by JobTracker to start RPC Server.

mapreduce.tasktracker.outofband.heartbeat

True

The task tracker sends an out-of-band heartbeat on task completion to improve latency. Set this value to false to disable this behavior.

webinterface.private.actions

False

By default, jobs cannot be killed from the JobTracker's web interface. Set this value to True to enable this behavior.

Icon

MapR recommends properly securing your interfaces before enabling this behavior.

maprfs.openfid2.prefetch.bytes

0

Expert: number of shuffle bytes to be prefetched by reduce task

mapr.localoutput.dir

output

The path for map output files on shuffle volume.

mapr.localspill.dir

spill

The path for local spill files on shuffle volume.

mapreduce.jobtracker.node.labels.file

 

The file that specifies the labels to apply to the nodes in the cluster.

mapreduce.jobtracker.node.labels.monitor.interval

120000

Specifies a time interval in milliseconds. The node labels file is polled for changes every time this interval passes.

mapred.queue.<queue-name>.label

 

Specifies a label for the queue named in the <queue-name> placeholder.

mapred.queue.<queue-name>.label.policy

 

Specifies a policy for the label applied to the queue named in the <queue-name> placeholder. The policy controls the interaction between the queue label and the job label:

  • PREFER_QUEUE — always use label set on queue
  • PREFER_JOB — always use label set on job
  • AND (default) — job label AND node label
  • OR — job label OR node label
ipc.server.noncurrent.version.allowedtrueWhen set to true, this parameter allows IPC version mismatches. Set this parameter to false to prevent JobClient or TaskTracker code from releases prior to version 3.1 of the MapR distribution for Hadoop from running with a 3.1 JobTracker.

JobTracker Directories

When changing any parameters in this section, a JobTracker restart is required.

Volume path = mapred.system.dir/../

Parameter

Value

Description

mapred.system.dir

/var/mapr/cluster/mapred/jobTracker/system

The shared directory where MapReduce stores control files.

mapred.job.tracker.persist.jobstatus.dir

/var/mapr/cluster/mapred/jobTracker/jobsInfo

The directory where the job status information is persisted in a file system to be available after it drops of the memory queue and between JobTracker restarts.

mapreduce.jobtracker.staging.root.dir

/var/mapr/cluster/mapred/jobTracker/staging

The root of the staging area for users' job files In practice, this should be the directory where users' home directories are located (usually /user)

mapreduce.job.split.metainfo.maxsize

10000000

The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1.

mapreduce.maprfs.use.compression

True

Set this property's value to False to disable the use of MapR-FS compression for shuffle data by MapReduce.

mapred.jobtracker.retiredjobs.cache.size

1000

The number of retired job status to keep in the cache.

mapred.job.tracker.history.completed.location

/var/mapr/cluster/mapred/jobTracker/history/done

The completed job history files are stored at this single well known location. If nothing is specified, the files are stored at ${hadoop.job.history.location}/done in local filesystem.

hadoop.job.history.location

 

If job tracker is static the history files are stored in this single well known place on local filesystem. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history. History files are moved to mapred.jobtracker.history.completed.location which is on MapRFs JobTracker volume.

mapred.jobtracker.jobhistory.lru.cache.size

5

The number of job history files loaded in memory. The jobs are loaded when they are first accessed. The cache is cleared based on LRU.

JobTracker Recovery

When changing any parameters in this section, a JobTracker restart is required.

Parameter

Value

Description

mapreduce.jobtracker.recovery.dir

/var/mapr/cluster/mapred/jobTracker/recovery

Recovery Directory. Stores list of known TaskTrackers.

mapreduce.jobtracker.recovery.maxtime

120

Maximum time in seconds JobTracker should stay in recovery mode.

mapreduce.jobtracker.split.metainfo.maxsize

10000000

This property's value sets the maximum permissible size of the split metainfo file. The JobTracker does not attempt to read split metainfo files larger than this value.

mapred.jobtracker.restart.recover

true

"true" to enable (job) recovery upon restart, "false" to start afresh

mapreduce.jobtracker.recovery.job.initialization.maxtime

480

this property's value specifies the maximum time in seconds that the JobTracker waits to initialize jobs before starting recovery. This property's default value is equal to the value of the mapreduce.jobtracker.recovery.maxtime property.

Enable Fair Scheduler

When changing any parameters in this section, a JobTracker restart is required.

Parameter

Value

Description

mapred.fairscheduler.allocation.file

conf/pools.xml

 

mapred.jobtracker.taskScheduler

org.apache.hadoop.mapred.FairScheduler

The class responsible for task scheduling.

mapred.fairscheduler.assignmultiple

true

 

mapred.fairscheduler.eventlog.enabled

false

Enable scheduler logging in ${HADOOP_LOG_DIR}/fairscheduler/

mapred.fairscheduler.smalljob.schedule.enable

True

Set this property's value to False to disable fast scheduling for small jobs in FairScheduler. TaskTrackers can reserve an ephemeral slot for small jobs when the cluster is under load.

mapred.fairscheduler.smalljob.max.maps

10

Small job definition. Max number of maps allowed in small job.

mapred.fairscheduler.smalljob.max.reducers

10

Small job definition. Max number of reducers allowed in small job.

mapred.fairscheduler.smalljob.max.inputsize

10737418240

Small job definition. Max input size in bytes allowed for a small job. Default is 10GB.

mapred.fairscheduler.smalljob.max.reducer.inputsize

1073741824

Small job definition. Max estimated input size for a reducer allowed in small job. Default is 1GB per reducer.

mapred.cluster.ephemeral.tasks.memory.limit.mb

200

Small job definition. Max memory in mbytes reserved for an ephermal slot. Default is 200mb. This value must be same on JobTracker and TaskTracker nodes.

TaskTracker Configuration

When changing any parameters in this section, a TaskTracker restart is required.

Icon

When mapreduce.tasktracker.prefetch.maptasks is greater than 0, you must disable Fair Scheduler with preemption and label-based job placement.

Parameter

Value

Description

mapred.tasktracker.map.tasks.maximum

-1

The maximum number of map task slots to run simultaneously. The default value of -1 specifies that the number of map task slots is based on the total amount of memory reserved for MapReduce by the Warden. Of the memory available for MapReduce (not counting the memory reserved for ephemeral slots), 40% is allocated to map tasks. That total amount of memory is divided by the value of the mapred.maptask.memory.default parameter to determine the total number of map task slots on this node. You can also specify a formula using the following variables:

  • CPUS - The number of CPUs on the node.
  • DISKS - The number of disks on the node.
  • MEM - The amount of memory (in MB) reserved for MapReduce tasks by the Warden.
    You can assemble these variables with the syntax CONDITIONAL ? TRUE : FALSE. For example, the expression 2*CPUS < DISKS ? 2*CPUS : DISKS results in 2*CPUS slots when there are more disks on the node than twice the number of cores, and DISKS slots otherwise.

mapreduce.tasktracker.prefetch.maptasks

0.0

The proportion of map tasks that can be scheduled in advance (prefetched) on a TaskTracker. The number is given as a ratio of prefetched tasks to the total number of map slots. For example, 0.25 means the number of prefetched tasks = 25% of the total number of map slots. The default is 0.0, which means no prefetched tasks can be scheduled.

mapreduce.tasktracker.reserved.physicalmemory.mb.low

0.8

This property's value sets the target memory usage level when the TaskTracker kills tasks to reduce total memory usage. This property's value represents a percentage of the amount in the mapreduce.tasktracker.reserved.physicalmemory.mb value.

mapreduce.tasktracker.task.slowlaunch

False

Set this property's value to True to wait after each task launch for nodes running critical services like CLDB, JobTracker, and ZooKeeper.

mapreduce.tasktracker.volume.healthcheck.interval

60000

This property's value defines the frequency in milliseconds that the TaskTracker checks the Mapreduce volume defined in the ${mapr.localvolumes.path}/mapred/ property.

mapreduce.use.maprfs

True

Use MapR-FS for shuffle and sort/merge.

mapred.userlog.retain.hours

24

This property's value specifies the maximum time, in hours, to retain the user-logs after job completion.

mapred.userlog.retain.hours.max168This property's value specifies the highest legal value for the mapred.userlog.retain.hours property. When a user specifies a value for mapred.userlog.retain.hours in excess of the value of the mapred.userlog.retain.hours.max property, that value is ignored and the value of the mapred.userlog.retain.hours.max property is used instead.

mapred.user.jobconf.limit

5242880

The maximum allowed size of the user jobconf. The default is set to 5 MB.

mapred.userlog.limit.kb

0

Deprecated: The maximum size of user-logs of each task in KB. 0 disables the cap.

mapreduce.use.fastreduce

False

Expert: Merge map outputs without copying.

mapred.tasktracker.reduce.tasks.maximum

-1

The maximum number of reduce task slots to run simultaneously. The default value of -1 specifies that the number of reduce task slots is based on the total amount of memory reserved for MapReduce by the Warden. Of the memory available for MapReduce (not counting the memory reserved for ephemeral slots), 60% is allocated to reduce tasks. That total amount of memory is divided by the value of the mapred.reducetask.memory.default parameter to determine the total number of reduce task slots on this node. You can also specify a formula using the following variables:

  • CPUS - The number of CPUs on the node.
  • DISKS - The number of disks on the node.
  • MEM - The amount of memory reserved for MapReduce tasks by the Warden.
    You can assemble these variables with the syntax CONDITIONAL ? TRUE : FALSE. For example, the expression 2*CPUS < DISKS ? 2*CPUS : DISKS results in 2*CPUS slots when there are more disks on the node than twice the number of cores, and DISKS slots otherwise.

mapred.tasktracker.ephemeral.tasks.maximum

1

Reserved slot for small job scheduling

mapred.tasktracker.ephemeral.tasks.timeout

10000

Maximum time in milliseconds a task is allowed to occupy ephemeral slot

mapred.tasktracker.ephemeral.tasks.ulimit

4294967296

Ulimit (bytes) on all tasks scheduled on an ephemeral slot

mapreduce.tasktracker.reserved.physicalmemory.mb

 

Maximum phyiscal memory TaskTracker should reserve for mapreduce tasks.
If tasks use more than the limit, task using maximum memory will be killed.
Expert only: Set this value only if TaskTracker should use a certain amount of memory
for mapreduce tasks. In MapR Distro warden figures this number based
on services configured on a node.
Setting mapreduce.tasktracker.reserved.physicalmemory.mb to -1 will disable
physical memory accounting and task management.

mapred.tasktracker.expiry.interval

600000

Expert: This property's value specifies a time interval in milliseconds. After this interval expires without any heartbeats sent, a TaskTracker is marked lost.

mapreduce.tasktracker.heapbased.memory.management

false

Expert only: If the admin wants to prevent swapping by not launching too many tasks,
use this option. Task's memory usage is based on max java heap size (-Xmx).
By default, -Xmx will be computed by the TaskTracker based on slots and memory reserved for mapreduce tasks.
See mapred.map.child.java.opts/mapred.reduce.child.java.opts.

mapreduce.tasktracker.jvm.idle.time

10000

If JVM is idle for more than mapreduce.tasktracker.jvm.idle.time (milliseconds)
TaskTracker will kill it.

mapred.max.tracker.failures

4

The number of task failures on a TaskTracker of a given job after which new tasks of that job aren't assigned to it.

mapred.max.tracker.blacklists

4

The number of blacklists for a TaskTracker by various jobs after which the TaskTracker could be blacklisted across all jobs. The TaskTracker will be given tasks later (after a day). The TaskTracker will become healthy after a restart.

mapred.task.tracker.http.address

0.0.0.0:50060

This property's value specifies the HTTP server address and port for the TaskTracker. Specify 0 as the port to make the server start on a free port.

mapred.task.tracker.report.address

127.0.0.1:0

The IP address and port that TaskTrackeer server listens on. Since it is only connected to by the tasks, it uses the local interface. EXPERT ONLY. Only change this value if your host does not have a loopback interface.

mapreduce.tasktracker.group

mapr

Expert: Group to which TaskTracker belongs. If LinuxTaskController is configured via the mapreduce.tasktracker.taskcontroller value, the group owner of the task-controller binary $HADOOP_HOME/bin/platform/bin/task-controller must be same as this group.

mapred.tasktracker.task-controller.config.overwrite

True

The LinuxTaskController needs a configuration file set at $HADOOP_HOME/conf/taskcontroller.cfg. The configuration file takes the following parameters:

  • mapred.local.dir = Local dir used by TaskTracker, taken from mapred-site.xml.
  • hadoop.log.dir = hadoop log dir, taken from system properties of the TaskTracker process
  • mapreduce.tasktracker.group = groups allowed to run TaskTracker see 'mapreduce.tasktracker.group'
  • min.user.id = Don't allow any user below this uid to launch a task.
  • banned.users = Users who are not allowed to launch any tasks.
  • If set to true, TaskTracker will always overwrite config file with default values as
  • min.user.id = -1(check disabled), banned.users = bin, mapreduce.tasktracker.group = root
    To disable this configuration and use a custom configuration, set this property's value to False and restart the TaskTracker.

mapred.tasktracker.indexcache.mb

10

This property's value specifies the maximum amount of memory allocated by the TaskTracker for the index cache. The index cache is used when the TaskTracker serves map outputs to reducers.

mapred.tasktracker.instrumentation

org.apache.hadoop.mapred.TaskTrackerMetricsInst

Expert: The instrumentation class to associate with each TaskTracker.

mapred.task.tracker.task-controller

org.apache.hadoop.mapred.LinuxTaskController

This property's value specifies the TaskController that launches and manages task execution.

mapred.tasktracker.taskmemorymanager.killtask.maxRSS

False

Set this property's value to True to kill tasks that are using maximum memory when the total number of MapReduce tasks exceeds the limit specified in the TaskTracker's mapreduce.tasktracker.reserved.physicalmemory.mb property. Tasks are killed in most-recently-launched order.

mapred.tasktracker.taskmemorymanager.monitoring-interval

3000

This property's value specifies an interval in milliseconds that TaskTracker waits between monitoring the memory usage of tasks. This property is only used when tasks memory management is enabled by setting the property mapred.tasktracker.tasks.maxmemory to True.

mapred.tasktracker.tasks.sleeptime-before-sigkill

5000

This property's value sets the time in milliseconds that the TaskTracker waits before sending a SIGKILL to a process after it has been sent a SIGTERM.

mapred.temp.dir

${hadoop.tmp.dir}/mapred/temp

A shared directory for temporary files.

mapreduce.cluster.map.userlog.retain-size

-1

This property's value specifies the number of bytes to retain from map task logs. The default value of -1 disables this feature.

mapreduce.cluster.reduce.userlog.retain-size

-1

This property's value specifies the number of bytes to retain from reduce task logs. The default value of -1 disables this feature.

mapreduce.heartbeat.10000

100000

This property's value specifies a heartbeat time in milliseconds for a medium cluster of 1001 to 10000 nodes. Scales linearly between 10s - 100s.

mapreduce.heartbeat.1000

10000

This property's value specifies a heartbeat time in milliseconds for a medium cluster of 101 to 1000 nodes. Scales linearly between 1s - 10s.

mapreduce.heartbeat.100

1000

This property's value specifies a heartbeat time in milliseconds for a medium cluster of 11 to 100 nodes. Scales linearly between 300ms - 1s.

mapreduce.heartbeat.10

300

This property's value specifies a heartbeat time in milliseconds for a medium cluster of 1 to 10 nodes.

mapreduce.job.complete.cancel.delegation.tokens

True

Set this property's value to False to prevent unregister or cancel delegation tokens from renewing.

mapreduce.jobtracker.inline.setup.cleanup

False

Set this property's value to True to make the JobTracker attempt to set up and clean up the job by itself or do it in setup/cleanup task.

Job Configuration

Set these values on the node from which you plan to submit jobs, before submitting the jobs. If you are using Hadoop examples, you can set these parameters from the command line. Example:

When you submit a job, the JobClient creates job.xml by reading parameters from the following files in the following order:

  1. mapred-default.xml
  2. The local mapred-site.xml - overrides identical parameters in mapred-default.xml
  3. Any settings in the job code itself - overrides identical parameters in mapred-site.xml

Parameter

Value

Description

keep.failed.task.files

false

Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. It also prevents the map outputs from being erased from the reduce directory as they are consumed.

mapred.job.reuse.jvm.num.tasks

-1

How many tasks to run per jvm. If set to -1, there is no limit.

mapred.map.tasks.speculative.execution

true

If true, then multiple instances of some map tasks may be executed in parallel.

mapred.reduce.tasks.speculative.execution

true

If true, then multiple instances of some reduce tasks may be executed in parallel.

mapred.reduce.tasks

1

The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when the value of the mapred.job.tracker property is local.

mapred.job.map.memory.physical.mb

 

Maximum physical memory limit for map task of this job. If limit is exceeded task attempt will be FAILED.

mapred.job.reduce.memory.physical.mb

 

Maximum physical memory limit for reduce task of this job. If limit is exceeded task attempt will be FAILED.

mapreduce.task.classpath.user.precedence

false

Set to true if user wants to set different classpath.

mapred.max.maps.per.node

-1

Per-node limit on running map tasks for the job. A value of -1 signifies no limit.

mapred.max.reduces.per.node

-1

Per-node limit on running reduce tasks for the job. A value of -1 signifies no limit.

mapred.running.map.limit

-1

Cluster-wide limit on running map tasks for the job. A value of -1 signifies no limit.

mapred.running.reduce.limit

-1

Cluster-wide limit on running reduce tasks for the job. A value of -1 signifies no limit.

mapreduce.tasktracker.cache.local.numberdirectories

10000

This property's value sets the maximum number of subdirectories to create in a given distributed cache store. Cache items in excess of this limit are expunged whether or not the total size threshold is exceeded.

mapred.reduce.child.java.opts

-XX:ErrorFile=/opt/cores/mapreduce_java_error%p.log

Java opts for the reduce tasks. MapR Default heapsize (-Xmx) is determined by memory reserved for mapreduce at TaskTracker. Reduce task is given more memory than map task. Default memory for a reduce task = (Total Memory reserved for mapreduce) * (2*#reduceslots / (#mapslots + 2*#reduceslots))

mapred.reduce.child.ulimit

 

 

io.sort.factor

256

The number of streams to merge simultaneously during file sorting. The value of this property determines the number of open file handles.

io.sort.mb

380

This value sets the size, in megabytes, of the memory buffer that holds map outputs before writing the final map outputs. Lower values for this property increases the chance of spills. Recommended practice is to set this value to 1.5 times the average size of a map output.

io.sort.record.percent

0.17

The percentage of the memory buffer specified by the io.sort.mb property that is dedicated to tracking record boundaries. The maximum number of records that the collection thread can collect before blocking is one-fourth of (io.sort.mb) x (io.sort.record.percent).

io.sort.spill.percent

0.99

This property's value sets the soft limit for either the buffer or record collection buffers. Threads that reach the soft limit begin to spill the contents to disk in the background. Note that this does not imply any chunking of data to the spill. Do not reduce this value below 0.5.

mapred.reduce.slowstart.completed.maps

0.95

Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job.

mapreduce.reduce.input.limit

-1

The limit on the input size of the reduce. If the estimated
input size of the reduce is greater than this value, job is failed. A
value of -1 means that there is no limit set.

mapred.reduce.parallel.copies

12

The default number of parallel transfers run by reduce during the copy(shuffle) phase.

jobclient.completion.poll.interval

5000

This property's value specifies the JobClient's polling frequency in milliseconds to the JobTracker for updates about job status. Reduce this value for faster tests on single node systems. Adjusting this value on production clusters may result in undesired client-server traffic.

jobclient.output.filter

FAILED

This property's value specifies the filter that controls the output of the task's userlogs that are sent to the JobClient's console. Legal values are:

  • NONE
  • KILLED
  • FAILED
  • SUCCEEDED
  • ALL

jobclient.progress.monitor.poll.interval

1000

This property's value specifies the JobClient's status reporting frequency in milliseconds to the console and checking for job completion.

job.end.notification.url

http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus

This property's value specifies the URL to call at job completion to report the job's end status. Only two variables are legal in the URL, $jobId and $jobStatus. When present, these variables are replaced by their respective values.

job.end.retry.attempts

0

This property's value specifies the maximum number of times that Hadoop attempts to contact the notification URL.

job.end.retry.interval

30000

This property's value specifies the interval in milliseconds between attempts to contact the notification URL.

keep.failed.task.files

False

Set this property's value to True to keep files for failed tasks. Because this storage is not automatically reclaimed by the system, keep files only for jobs that are failing. Setting this property's value to True also keeps map outputs in the reduce directory as the map outputs are consumed instead of deleting the map outputs on consumption.

local.cache.size

10737418240

This property's value specifies the number of bytes allocated to each local TaskTracker directory to store Distributed Cache data.

mapr.centrallog.dir

logs

This property's value specifies the relative path under a local volume path that points to the central log location, ${mapr.localvolumes.path}/<hostname>/${mapr.centrallog.dir}.

mapr.localvolumes.path

/var/mapr/local

The path for local volumes.

map.sort.class

org.apache.hadoop.util.QuickSort

The default sort class for sorting keys.

tasktracker.http.threads

2

The number of worker threads that for the HTTP server.

topology.node.switch.mapping.impl

org.apache.hadoop.net.ScriptBasedMapping

The default implementation of the DNSToSwitchMapping. It invokes a script specified in the topology.script.file.name property to resolve node names. If no value is set for the topology.script.file.name property, the default value of DEFAULT_RACK is returned for all node names.

topology.script.number.args

100

The max number of arguments that the script configured with the topology.script.file.name runs with. Each argument is an IP address.

mapr.task.diagnostics.enabled

False

Set this property's value to True to run the MapR diagnostics script before killing an unresponsive task attempt.

mapred.acls.enabled

False

This property's value specifies whether or not to check ACLs for user authorization during various queue and job level operations. Set this property's value to True to enable access control checks made by the JobTracker and TaskTracker when users request queue and job operations using Map/Reduce APIs, RPCs, the console, or the web user interfaces.

mapred.child.oom_adj

10

This property's value specifies the adjustment to the out-of-memory value for the Linux-specific out-of-memory killer. Legal values are 0-15.

mapred.child.renice

10

This property's value specifies an integer from 0 to 19 for use by the Linux nice}} utility.

mapred.child.taskset

True

Set this property's value to False to prevent running the job in a taskset. See the manual page for taskset(1) for more information.

mapred.child.tmp

./tmp

This property's value sets the location of the temporary directory for map and reduce tasks. Set this value to an absolute path to directly assign the directory. Relative paths are located under the task's working directory. Java tasks execute with the option -Djava.io.tmpdir=absolute path of the tmp dir. Pipes and streaming are set with environment variable TMPDIR=absolute path of the tmp dir.

mapred.cluster.ephemeral.tasks.memory.limit.mb

200

This property's value specifies the maximum size in megabytes for small jobs. This value is reserved in memory for an ephemeral slot. JobTracker and TaskTracker nodes must set this property to the same value.

mapred.cluster.map.memory.mb

-1

This property's value sets the virtual memory size of a single map slot in the Map-Reduce framework used by the scheduler. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.job.map.memory.mb, to the limit specified by the value of mapred.cluster.max.map.memory.mb. The default value of -1 disables the feature. Set this value to a useful memory size to enable the feature.

mapred.cluster.max.map.memory.mb

-1

This property's value sets the virtual memory size of a single map task launched by the Map-Reduce framework used by the scheduler. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.job.map.memory.mb, to the limit specified by the value of mapred.cluster.max.map.memory.mb. The default value of -1 disables the feature. Set this value to a useful memory size to enable the feature.

mapred.cluster.max.reduce.memory.mb

-1

This property's value sets the virtual memory size of a single reduce task launched by the Map-Reduce framework used by the scheduler. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.job.reduce.memory.mb, to the limit specified by the value of mapred.cluster.max.reduce.memory.mb. The default value of -1 disables the feature. Set this value to a useful memory size to enable the feature.

mapred.cluster.reduce.memory.mb

-1

This property's value sets the virtual memory size of a single reduce slot in the Map-Reduce framework used by the scheduler. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.job.reduce.memory.mb, to the limit specified by the value of mapred.cluster.max.reduce.memory.mb. The default value of -1 disables the feature. Set this value to a useful memory size to enable the feature.

mapred.compress.map.output

False

Set this property's value to True to compress map outputs with SequenceFile compresison before sending the outputs over the network.

mapred.fairscheduler.assignmultiple

True

Set this property's value to False to prevent the FairScheduler from assigning multiple tasks.

mapred.fairscheduler.eventlog.enabled

False

Set this property's value to True to enable scheduler logging in {{${HADOOP_LOG_DIR}/fairscheduler/

mapred.fairscheduler.smalljob.max.inputsize

10737418240

This property's value specifies the maximum size, in bytes, that defines a small job.

mapred.fairscheduler.smalljob.max.maps

10

This property's value specifies the maximum number of maps allowed in a small job.

mapred.fairscheduler.smalljob.max.reducer.inputsize

1073741824

This property's value specifies the maximum estimated input size, in bytes, for a reducer in a small job.

mapred.fairscheduler.smalljob.max.reducers

10

This property's value specifies the maximum number of reducers allowed in a small job.

mapred.healthChecker.interval

60000

This property's value sets the frequency, in milliseconds, that the node health script runs.

mapred.healthChecker.script.timeout

600000

This property's value sets the frequency, in milliseconds, after which the node script is killed for being unresponsive and reported as failed.

mapred.inmem.merge.threshold

1000

When a number of files equal to this property's value accumulate, the in-memory merge triggers and spills to disk. Set this property's value to zero or less to force merges and spills to trigger solely on RAMFS memory consumption.

mapred.job.map.memory.mb

-1

This property's value sets the virtual memory size of a single map task for the job. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.cluster.map.memory.mb, to the limit specified by the value of mapred.cluster.max.map.memory.mb. The default value of -1 disables the feature if the value of the mapred.cluster.map.memory.mgb property is also -1. Set this value to a useful memory size to enable the feature.

mapred.job.queue.name

default

This property's value specifies the queue a job is submitted to. This property's value must match the name of a queue defined in mapred.queue.names for the system. The ACL setup for the queue must allow the current user to submit a job to the queue.

mapred.job.reduce.input.buffer.percent

0

This property's value specifies the percentage of memory relative to the maximum heap size. After the shuffle, remaining map outputs in memory must occupy less memory than this threshold value before reduce begins.

mapred.job.reduce.memory.mb

-1

This property's value sets the virtual memory size of a single reduce task for the job. If the scheduler supports this feature, a job can ask for multiple slots for a single map task via mapred.cluster.reduce.memory.mb, to the limit specified by the value of mapred.cluster.max.reduce.memory.mb. The default value of -1 disables the feature if the value of the mapred.cluster.map.memory.mgb property is also -1. Set this value to a useful memory size to enable the feature.

mapred.job.reuse.jvm.num.tasks

-1

This property's value sets the number of tasks to run on each JVM. The default of -1 sets no limit.

mapred.job.shuffle.input.buffer.percent

0.7

This property's value sets the percentage of memory allocated from the maximum heap size to storing map outputs during the shuffle.

mapred.job.shuffle.merge.percent

0.66

This property's value sets a percentage of the total memory allocated to storing map outputs in mapred.job.shuffle.input.buffer.percent. When memory storage for map outputs reaches this percentage, an in-memory merge triggers.

mapred.job.tracker.handler.count

10

This property's value sets the number of server threads for the JobTracker. As a best practice, set this value to approximately 4% of the number of TaskTracker nodes.

mapred.job.tracker.history.completed.location

/var/mapr/cluster/mapred/jobTracker/history/done

This property's value sets a location to store completed job history files. When this property has no value specified, completed job files are stored at ${hadoop.job.history.location}/done in the local filesystem.

mapred.job.tracker.http.address

0.0.0.0:50030

This property's value specifies the HTTP server address and port for the JobTracker. Specify 0 as the port to make the server start on a free port.

mapred.jobtracker.instrumentation

org.apache.hadoop.mapred.JobTrackerMetricsInst

Expert: The instrumentation class to associate with each JobTracker.

mapred.jobtracker.job.history.block.size

3145728

This property's value sets the block size of the job history file. Dumping job history to disk is important because job recovery uses the job history.

mapred.jobtracker.jobhistory.lru.cache.size

5

This property's value specifies the number of job history files to load in memory. The jobs are loaded when they are first accessed. The cache is cleared based on LRU.

mapred.job.tracker

maprfs:///

JobTracker address ip:port or use uri maprfs:/// for default cluster or maprfs:///mapr/san_jose_cluster1 to connect 'san_jose_cluster1' cluster. ""local"" for standalone mode.

mapred.jobtracker.maxtasks.per.job

-1

Set this property's value to any positive integer to set the maximum number of tasks for a single job. The default value of -1 indicates that there is no maximum.

mapred.job.tracker.persist.jobstatus.active

False

Set this property's value to True to enable persistence of job status information.

mapred.job.tracker.persist.jobstatus.dir

/var/mapr/cluster/mapred/jobTracker/jobsInfo

This property's value specifies the directory where job status information persists after dropping out of the memory queue between JobTracker restarts.

mapred.job.tracker.persist.jobstatus.hours

0

This property's value specifies job status information persistence time in hours. Persistent job status information is available after the information drops out of the memory queue and between JobTracker restarts. The default value of zero disables job status information persistence.

mapred.jobtracker.port

9001

The IPC port on which the JobTracker listens.

mapred.jobtracker.restart.recover

True

Set this property's value to False to disable job recovery on restart.

mapred.jobtracker.retiredjobs.cache.size

1000

This property's value specifies the number of retired job statuses kept in the cache.

mapred.jobtracker.retirejob.check

30000

This property's value specifies the frequency interval used by the retire job thread to check for completed jobs.

mapred.line.input.format.linespermap

1

Number of lines per split in NLineInputFormat.

mapred.local.dir.minspacekill

0

This property's value specifies a threshold of free space in the directory specified by the mapred.local.dir property. When free space drops below this threshold, no more tasks are requested until all current tasks finish and clean up. When free space is below this threshold, running tasks are killed in the following order until free space is above the threshold:

  • Reduce tasks
  • All other tasks in reverse percent-completed order.

mapred.local.dir.minspacestart

0

This property's value specifies a free space threshold for the directory specified by mapred.local.dir. No tasks are requested while free space is below this threshold.

mapred.local.dir

/tmp/mapr-hadoop/mapred/local

This property's value specifies the directory where MapReduce localized job files. Localized job files are the job-related files downloaded by the TaskTracker and include the job configuration, job JAR file, and files added to the DistributedCache. Each task attempt has a dedicated subdirectory under the mapred.local.dir directory. Shared files are symbolically linked to those subdirectories.

mapred.map.child.java.opts

-XX:ErrorFile=/opt/cores/mapreduce_java_error%p.log

This property stores Java options for map tasks. When present, the @taskid@ symbol is replaced with the current TaskID. As an example, to enable verbose garbage collection logging to a file named for the taskid in /tmp and to set the heap maximum to 1GB, set this property to the value -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc.
The configuration variable mapred.{map/reduce}.child.ulimit controls the maximum virtual memory of the child processes.
In the MapR distribution for Hadoop, the default -Xmx is determined by memory reserved for mapreduce by the TaskTracker. Reduce tasks use memory than map tasks. The default memory for a map task follows the formula (Total Memory reserved for mapreduce) * (#mapslots/ (#mapslots + 1.3*#reduceslots)).

mapred.map.child.log.level

INFO

This property's value sets the logging level for the map task. The allowed levels are:

  • OFF
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE
  • ALL

mapred.map.max.attempts

4

Expert: This property's value sets the maximum number of attempts per map task.

mapred.map.output.compression.codec

org.apache.hadoop.io.compress.DefaultCodec

Specifies the compression codec to use to compress map outputs if compression of map outputs is enabled.

mapred.maptask.memory.default

800

When the value of the mapred.tasktracker.map.tasks.maximum parameter is -1, this parameter specifies a size in MB that is used to determine the default total number of map task slots on this node.

mapred.map.tasks

2

The default number of map tasks per job. Ignored when the value of the mapred.job.tracker property is local.

mapred.maxthreads.generate.mapoutput

1

Expert: Number of intra-map-task threads to sort and write the map output partitions.

mapred.maxthreads.partition.closer

1

Expert: Number of threads that asynchronously close or flush map output partitions.

mapred.merge.recordsBeforeProgress

10000

The number of records to process during a merge before sending a progress notification to the TaskTracker.

mapred.min.split.size

0

The minimum size chunk that map input should be split into. File formats with minimum split sizes take priority over this setting.

mapred.output.compress

False

Set this property's value to True to compress job outputs.

mapred.output.compression.codec

org.apache.hadoop.io.compress.DefaultCodec

When job output compression is enabled, this property's value specifies the compression codec.

mapred.output.compression.type

RECORD

When job outputs are compressed as SequenceFiles, this value's property specifies how to compress the job outputs. Legal values are:

  • NONE
  • RECORD
  • BLOCK

mapred.queue.default.state

RUNNING

This property's value defines the state of the default queue, which can be either STOPPED or RUNNING. This value can be changed at runtime.

mapred.queue.names

default

This property's value specifies a comma-separated list of the queues configured for this JobTracker. Jobs are added to queues and schedulers can configure different scheduling properties for the various queues. To configure a property for a queue, the name of the queue must match the name specified in this value. Queue properties that are common to all schedulers are configured here with the naming convention mapred.queue.$QUEUE-NAME.$PROPERTY-NAME.
The number of queues configured in this parameter can depend on the type of scheduler being used, as specified in mapred.jobtracker.taskScheduler. For example, the JobQueueTaskScheduler supports only a single queue, which is the default configured here. Verify that the schedule supports multiple queues before adding queues.

mapred.reduce.child.log.level

INFO

The logging level for the reduce task. The allowed levels are:

  • OFF
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE
  • ALL

mapred.reduce.copy.backoff

300

This property's value specifies the maximum amount of time in seconds a reducer spends on fetching one map output before declaring the fetch failed.

mapred.reduce.max.attempts

4

Expert: The maximum number of attempts per reduce task.

mapred.reducetask.memory.default

1500

When the value of the mapred.tasktracker.reduce.tasks.maximum parameter is -1, this parameter specifies a size in MB that is used to determine the default total number of reduce task slots on this node.

mapred.skip.attempts.to.start.skipping

2

This property's value specifies a number of task attempts. After that many task attempts, skip mode is active. While skip mode is active, the task reports the range of records which it will process next to the TaskTracker. With this record range, the TaskTracker is aware of which records are dubious and skips dubious records on further executions.

mapred.skip.map.auto.incr.proc.count

True

SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS increments after MapRunner invokes the map function. Set this property's value to False for applications that process records asynchronously or buffer input records. Such applications must increment this counter directly.

mapred.skip.map.max.skip.records

0

The number of acceptable skip records around the bad record, per bad record in the mapper. The number includes the bad record. The default value of 0 disables detection and skipping of bad records. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to prevent the framework from narrowing down the skipped range.

mapred.skip.reduce.auto.incr.proc.count

True

SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS increments after MapRunner invokes the reduce function. Set this property's value to False for applications that process records asynchronously or buffer input records. Such applications must increment this counter directly.

mapred.skip.reduce.max.skip.groups

0

The number of acceptable skip records around the bad record, per bad record in the reducer. The number includes the bad record. The default value of 0 disables detection and skipping of bad records. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to prevent the framework from narrowing down the skipped range.

mapred.submit.replication

10

This property's value specifies the replication level for submitted job files. As a best practice, set this value to approximately the square root of the number of nodes.

mapred.task.cache.levels

2

This property's value specifies the maximum level of the task cache. For example, if the level is 2, the tasks cached are at the host level and at the rack level.

mapred.task.calculate.resource.usage

True

Set this property's value to False to prevent the use of the ${mapreduce.tasktracker.resourcecalculatorplugin} parameter.

mapred.task.profile

False

Set this property's value to True to enable task profiling and the collection of profiler information by the system.

mapred.task.profile.maps

0-2

This property's value sets the ranges of map tasks to profile. This property is ignored when the value of the mapred.task.profile property is set to False.

mapred.task.profile.reduces

0-2

This property's value sets the ranges of reduce tasks to profile. This property is ignored when the value of the mapred.task.profile property is set to False.

mapred.task.timeout

600000

This property's value specifies a time in milliseconds after which a task terminates if the task does not perform any of the following:

  • reads an input
  • writes an output
  • updates its status string

mapred.tasktracker.dns.interface

default

This property's value specifies the name of the network interface that the TaskTracker reports its IP address from.

mapred.tasktracker.dns.nameserver

default

This property's value specifies the host name or IP address of the name server (DNS) that the TaskTracker uses to determine the JobTracker's hostname.

Oozie

Parameter

Value

Description

hadoop.proxyuser.root.hosts

*

Specifies the hosts that the superuser must connect from in order to act as another user. Specify the hosts as a comma-separatedlist of IP addresses or hostnames that are running Oozie servers.

hadoop.proxyuser.mapr.groups

mapr,staff

 

hadoop.proxyuser.root.groups

root

The superuser can act as any member of the listed groups.

  • No labels