This is documentation for MapR Version 5.0. You can also refer to MapR documentation for the latest release.

Skip to end of metadata
Go to start of metadata
MapR provides an express path (called ExpressLane) that works in conjunction with the Fair Scheduler. ExpressLane is for small MapReduce jobs to run when all slots are occupied by long tasks. Small jobs are only given this special treatment when the cluster is busy, and only if they meet the criteria specified by the following parameters in mapred-site.xml:

Parameter

Value

Description

mapred.fairscheduler.smalljob.schedule.enable

true

Enable small job fast scheduling inside fair scheduler. TaskTrackers should reserve a slot called ephemeral slot, which is used for smalljob if the cluster is busy.

mapred.fairscheduler.smalljob.max.maps

10

Small job definition. Max number of maps allowed in small job.

mapred.fairscheduler.smalljob.max.reducers

10

Small job definition. Max number of reducers allowed in small job.

mapred.fairscheduler.smalljob.max.inputsize

10737418240

Small job definition. Max input size in bytes allowed for a small job. Default is 10GB.

mapred.fairscheduler.smalljob.max.reducer.inputsize

1073741824

Small job definition. Max estimated input size for a reducer allowed in small job. Default is 1GB per reducer.

mapred.cluster.ephemeral.tasks.memory.limit.mb

200

Small job definition. Max memory in mbytes reserved for an ephermal slot. Default is 200mb. This value must be same on JobTracker and TaskTracker nodes.

MapReduce jobs that appear to fit the small job definition but are in fact larger than anticipated are killed and re-queued for normal execution.

  • No labels