This is documentation for MapR Version 5.0. You can also refer to MapR documentation for the latest release.

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The The hadoop-metrics.properties files  files direct MapR where to output service metric reports: to an output MRv1 job metrics and service metrics reports. You can output the reports to file (FileContext) or to , to Ganglia 3 3.1 (MapRGangliaContext31). A third context, NullContext, disables metrics. To direct metrics to an output file, comment out the lines pertaining to Ganglia and the NullContext; for the chosen service; to direct metrics to Ganglia, comment out the lines pertaining to the metrics file and the NullContext. See Service Metrics.There are two , or disable the output. 

There are two hadoop-metrics.properties files files:

  • /opt/mapr/hadoop/hadoop-<version>0.2.0/conf/hadoop-metrics.properties specifies  specifies output for standard Hadoop servicesMRv1 job metrics
  • /opt/mapr/conf/hadoop-metrics.properties specifies  specifies output for MapR-specific services

The following table describes the parameters for each service in the the hadoop-metrics.properties files files.

Parameter

Example Values

Description

<service>.class

  • org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
  • apache.hadoop.metrics.file.FileContext
  • com.mapr.fs.cldb.counters.MapRGangliaContext31

The class that implements the interface responsible for sending the service metrics to the appropriate handler. When implementing a class that sends metrics to Ganglia, set this property to the class name.

<service>.period

  • 10
  • 60

The interval between 2 service metrics data exports to the appropriate interface. This is independent of how often are the metrics updated in the framework.

<service>.fileName

/tmp/cldbmetrics.log

The path to the file where service metrics are exported when the cldb.class property is set to FileContext.

<service<service>.servers

localhost:8649

The location of the gmon or gmeta that is aggregating metrics for this instance of the service, when the cldb.class property is set to GangliaContext.

Info
By default, the CLDB and FileServer metrics are sent via unicast to the Ganglia gmon server running on localhost. To send the metrics directly to a Gmeta server, change the cldb.servers property to the hostname of the Gmeta server. To send the metrics to a multicast channel, change the cldb.servers property to the IP address of the multicast channel.

<service>.spoof

1

Specifies whether the metrics being sent out from the server should be spoofed as coming from another server. All our fileserver metrics are also on cldb, but to make it appear to end users as if these properties were emitted by fileserver host, we spoof the metrics to Ganglia using this property. Currently only used for the FileServer service.

Examples

...

Configure Metrics

  1. Edit the appropriate hadoop-metrics.properties file. 
    1. For the MapR-FS and the CLDB edit /opt/mapr/conf/hadoop-metrics.properties on all CLDB nodes.
    2. For MRv1 Metrics, edit /opt/mapr/hadoop/hadoop-0.2.0/conf/hadoop-metrics.properties on all metrics nodes.
  2. In the sections specific to the service:
    • Un-comment the lines pertaining to the context to which you wish the service to send metrics.
    • Comment out the lines pertaining to other contexts. 
  3. Restart the service.

Disable Metrics

Comment out the following parameters in the /opt/mapr/hadoop/hadoop-0.2.0/conf/hadoop-metrics.properties file to disable job metrics.

  • maprmepredvariant.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext
  • maprmepredvariant.period=10
  • maprmapred.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContextFinal
  • maprmapred.period=10
Info
For information on how to enable or disable MapR-FS and CLDB metrics, see service metrics.

Examples

The hadoop-metrics.properties files  files are organized into sections for each service that provides metrics. Each section is divided into subsections for the three contexts.

MRv1 Job Metrics Example

The following file disables MRv1 job metrics in the /opt/mapr/hadoop/hadoop-

...

0.2.0/conf/hadoop-metrics.properties file:

Code Block

# Configuration of the "dfs" context for null
dfs.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "dfs" context for file
#dfs.class=org.apache.hadoop.metrics.file.FileContext
#dfs.period=10
#dfs.fileName=/tmp/dfsmetrics.log

# Configuration of the "dfs" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
# dfs.period=10
# dfs.servers=localhost:8649


# Configuration of the "mapred" context for nulldefault
mapred.class=org.apache.hadoop.metrics.spi.NullContextMapRDefaultContext
mapred.period=30

# Configuration of the "mapred" context for file
#mapred.class=org.apache.hadoop.metrics.file.FileContext
#mapred.period=10
#mapred.fileName=/tmp/mrmetrics.log

# Configuration of the "mapred" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
# mapred.period=10
# mapred.servers=localhost:8649


# Configuration of the "jvm" context for null
#jvmjvm.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "jvm" context for file
#jvm.class=org.apache.hadoop.metrics.file.FileContext
#jvm.period=10
#jvm.fileName=/tmp/jvmmetrics.log

# Configuration of the "jvm" context for ganglia
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# jvm.period=10
# jvm.servers=localhost:8649


# Configuration of the "ugi" context for null
ugi.class=org.apache.hadoop.metrics.spi.NullContext


# Configuration of the "fairscheduler" context for null
#fairschedulerfairscheduler.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "fairscheduler" context for file
#fairscheduler.class=org.apache.hadoop.metrics.file.FileContext
#fairscheduler.period=10
#fairscheduler.fileName=/tmp/fairschedulermetrics.log

# Configuration of the "fairscheduler" context for ganglia
# fairscheduler.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# fairscheduler.period=10
# fairscheduler.servers=localhost:8649
#

#maprmepredvariant.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext
#maprmepredvariant.period=60
#maprmapred.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContextFinal
#maprmapred.period=60

MapR-FS and CLDB Metrics Example

The following example enables the MapRGangliaContext31 context for the MapR-FS and the CLDB in the /opt/mapr/conf/hadoop-metrics.properties file:

Code Block

###########################################################################################################################
                                                      hadoop-metrics.properties
###########################################################################################################################

#CLDB metrics config - Pick one out of null,file or ganglia.
#Uncomment all properties in null, file or ganglia context, to send cldb metrics to that context

# Configuration of the "cldb" context for null
#cldb.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
#cldb.period=10

# Configuration of the "cldb" context for file
#cldb.class=org.apache.hadoop.metrics.file.FileContext
#cldb.period=60
#cldb.fileName=/tmp/cldbmetrics.log

# Configuration of the "cldb" context for ganglia
cldb.class=com.mapr.fs.cldb.counters.MapRGangliaContext31
cldb.period=10
cldb.servers=localhost:8649
cldb.spoof=1

#FileServer metrics config - Pick one out of null,file or ganglia.
#Uncomment all properties in null, file or ganglia context, to send fileserver metrics to that context

# Configuration of the "fileserver" context for null
#fileserver.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
#fileserver.period=10

# Configuration of the "fileserver" context for file
#fileserver.class=org.apache.hadoop.metrics.file.FileContext
#fileserver.period=60
#fileserver.fileName=/tmp/fsmetrics.log

# Configuration of the "fileserver" context for ganglia
fileserver.class=com.mapr.fs.cldb.counters.MapRGangliaContext31
fileserver.period=37
fileserver.servers=localhost:8649
fileserver.spoof=1

################################################################################################################
maprmepredvariant.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext
maprmepredvariant.period=10
maprmapred.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContextFinal
maprmapred.period=10