This is documentation for MapR Version 5.0. You can also refer to MapR documentation for the latest release.

Skip to end of metadata
Go to start of metadata

You can use logs to view the status and analyze the execution of jobs and applications in a cluster.

To view the status or to access logs for running jobs or applications, you can use the user interface associated with the MapReduce or YARN framework. 

  • For MapReduce v1 jobs, access the JobTracker user interface from the MapR Control System (MCS) to view the status and logs for a particular job. 
  • For MapReduce v2 or non-MapReduce applications, access the ResourceManager user interface from the MCS to view the status and logs for a particular application.

For completed jobs and applications, the distributed nature of the MapReduce and YARN frameworks can make analyzing the execution of jobs and applications difficult because tasks and containers are scattered throughout the cluster. Without centralized or aggregated logging, you must manually access all the log files for a completed job or application by merging the log details for a particular job across multiple nodes in the cluster. With centralized or aggregated logging, you can can access all the logs for a completed job or application in a centralized location. However, the steps to access logs for completed jobs or applications differ based on the configured logging option. For more information, see Centralized Logging or YARN Log Aggregation.

Logging Options

In a MapR cluster, the logging option that you configure defines how the logs are stored and accessed.

  • Centralized logging. The logs are written to local volumes on the MapR-FS. 
  • YARN log aggregation. The logs are written to the local file system and then the container logs from each node are aggregated and stored on the MapR-FS.  
  • Local logging. The log files for each job or application are written to the local file system. This is the default behavior for MapReduce v1 jobs,  MapReduce v2 applications, and non-MapReduce applications
    • For MapReduce v1 jobs, logs are written to the following directory on the local filesystem: /opt/mapr/hadoop/hadoop-0.20.0/logs/userlogs 
    • For MapReduce v2 or other applications that run on YARN, the logs are written to the following directory on the local filesystem: /opt/mapr/hadoop/hadoop-2.x.x/logs/userlogs/

The logging options that you can choose from are determined by the type of jobs or applications that you run:

Type of Job or ApplicationAvailable Logging Options
MapReduce v2
  • Centralized logging
  • YARN log aggregation
  • Local logging (default)

YARN applications (non- MapReduce)

  • YARN log aggregation
  • Local logging (default)
MapReduce v1
  • Centralized logging
  • Local logging (default)

If you enable centralized logging for MapReduce v2, the MapReduce applications will use centralized logging, while the other YARN applications in the cluster will use local logging.

Icon

Select a logging option that stores the logs on the MapR-FS for the following reasons:

  • Prevent job or application failures due to a lack of space on the local file system for logs.
  • Prevent the loss or inaccessibility of logs due to node failure. Logs stored in a local volume are two-way replicated.

For information about how to configure centralized logging, see Centralized Logging. For information about how to configure YARN log aggregation, see YARN Log Aggregation

 

  • No labels