This is documentation for MapR Version 5.0. You can also refer to MapR documentation for the latest release.

Skip to end of metadata
Go to start of metadata

The following sections describe the properties that you can configure for the ResourceManager. The default values for these properties are defined in the yarn-default.xml or by MapR. You can configure  overrides to the default by adding to or editing the properties in yarn-site.xml.

This topics includes the following sections:

ResourceManager Failover Properties 

The following table describes the configuration properties for ResourceManager failover:

 

PropertyDescription
yarn.resourcemanager.ha.custom-ha-enabledWhen yarn.client.failover-proxy-provider is set to org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider, this property must true.
The default, set by configure.sh in yarn-site.xml when the cluster uses zero configuration failover for the ResourceManager, is true.
yarn.resourcemanager.ha.enabled

Enables high availability for the ResourceManager. The default, set by MapR in the yarn-site.xml, is true.

This property must be set to true for failover to occur.

yarn.resourcemanager.ha.automatic-failover.enabled

When yarn.resourcemanager.ha.enabled is true, this property enables the ResourceManager to automatically failover.

The default, set in yarn-default.xml, is true.

yarn.resourcemanager.ha.automatic-failover.embedded

When yarn.resourcemanager.ha.enabled is true, this property enables the ResourceManager to use the embedded automatic failover.  

The default, set in yarn-default.xml, is true.

yarn.resourcemanager.cluster-id

Specifies the cluster that the ResourceManager belongs to.

This value is originally set by configure.sh in the yarn-site.xml and the value is required for failover to occur.

yarn.resourcemanager.ha.rm-idsThe ResourceManager service ID. Configure.sh adds this property to each node with the ResourceManager role.
yarn.resourcemanager.ha.id

Specifies the serviceID of the ResourceManager on the current node.

yarn.resourcemanager.zk-address

Specifies the zookeeper quorum that the ResourceManager belongs to.

This value is originally set by configure.sh in the yarn-site.xml when you configure failover.

yarn.client.failover-proxy-provider

Specifies the ResourceManager failover implementation used by clients, ApplicationMasters, and NodeManagers.

 configure.sh sets this value based on the type of failover that you configure.

  • For automatic or manual failover, configure.sh sets this value to org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 
  • For zero configuration failover, configure.sh sets this value to org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider

This value is set by configure.sh in yarn-site.xml when you configure failover. Otherwise, the default, set in yarn-default.xml is org.apache.hadoop.yarn.client.DefaultFailoverProxyProvider.

yarn.resourcemanager.scheduler.address[.<serviceID>]

The address of the scheduler interface

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.resourcemanager.resource-tracker.address[.<serviceID>]

The address of the resource tracker interface. ResourceManager listens for container requests and heartbeats from the NodeManagers on this port.

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.resourcemanager.address[.<serviceID>]

The address of the client interface. The ResourceManager listens for client requests on this port.

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.resourcemanager.admin.address[.<serviceID>]

The address of the administrative interface. ResourceManager listens for administrative requests from the yarn rmadmin command on this port.

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.resourcemanager.webapp.address[.<serviceID>]

The address of the ResourceManager web UI.

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.resourcemanager.webapp.https.address[.<serviceID>]

The address of the secure ResourceManager web UI.

This value, including the serviceID, is set by configure.sh in yarn-site.xml when you configure manual or automatic failover.

For zero configuration failover, this property in not needed unless you have configure custom port values. When you specify the custom port number, the serviceID is not required.

yarn.client.failover-max-attempts

The max number of times FailoverProxyProvider should attempt failover.

The default is -1.

yarn.client.failover-sleep-base-ms

The sleep base (in milliseconds) to be used for calculating the exponential delay between failovers.

The value defaults to the value set by yarn.resourcemanager.connect.retry-interval.ms, which is 30000 ms.

yarn.client.failover-sleep-max-ms

The maximum sleep time (in milliseconds) between failovers.

The value defaults to the value set by yarn.resourcemanager.connect.retry-interval.ms, which is 30000 ms.

yarn.client.failover-retries

The number of times a client attempts to reconnect to a ResourceManager.

The default, set in yarn-default.xml, is 0 (infinite).

yarn.client.failover-retries-on-socket-timeouts

The number of times a client attempts to reconnect to a ResourceManager on socket timeouts.

The default, set in yarn-default.xml, is 0 (infinite).

ResourceManager Recovery Properties

PropertyDescription
yarn.resourcemanager.recovery.enabled

Enables the Resourcemanager to recovery based on the information in the ResourceManager state store.

The default, set by configure.sh, is true.

yarn.resourcemanager.am.max-attempts

The maximum number of application attempts. This is a global setting for all ApplicationMaster nodes.

You can configure an individual maximum number of application attempts for each ApplicationMaster node, but this property sets a global upper bound that overrides the individual node configuration.

The default, set in yarn-default.xml, is 2.

mapreduce.am.max-attempts

The maximum number of MapReduce application attempts. If this value is larger than the value set by the ResourceManager, the ResourceManager value will override this value. The default number is set to 2, to allow at least one retry for AM. This property is set in mapred-default.xml.
yarn.resourcemanager.fs.state-store.uri

URI pointing to the location of the FileSystem path where the ResourceManager state is stored.

The default value is configured to the path for the ResourceManager volume (/var/mapr/cluster/yarn/rm/system).

 If the FileSystem name is not provided, the system uses the value specified in the fs.default.name specified in the core-site.xml file.

yarn.resourcemanager.fs.state-store.retry-policy-spec

Specifies the retry policy for the MapR-FS client.

This policy is specified in pairs of values for the sleep time, in milliseconds, and number of retries.

Each pair is enclosed in parentheses, such as (1000,20), (2000,30).

The previous example sleeps for 1000 milliseconds for twenty retries, then thirty more retries 2000 milliseconds apart.

The default, set in yarn-default.xml, is (2000,500).

yarn.resourcemanager.store.class

The class name of the state-store to be used for saving application/attempt state and the credentials.

The available state-store implementations are 
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore,
a ZooKeeper based state-store implementation, and 
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
a state-store implementation based on MapR-FS.

The default, yarn-default.xml, is org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.

yarn.resourcemanager.state-store.max-completed-applications

The maximum number of completed applications that the state store retains, which is a number less than or equal to 
${yarn.resourcemanager.max-completed-applications}.

The default value is 10000. This setting ensures that the applications kept in the state store are consistent with the applications in ResourceManager memory.

Any value larger than 
${yarn.resourcemanager.max-completed-applications} is reset to the default.

The value of this property affects ResourceManager recovery performance.Typically, a smaller value optimizes performance for recovery.

yarn.resourcemanager.zk-address

A comma-separated list of Host:Port pairs. Each corresponds to a ZooKeeper server, 
such as 127.0.0.1:5181,127.0.0.1:5181,127.0.0.1:5181.

These hosts are used by the ResourceManager to store state.

yarn.resourcemanager.zk-state-store.parent-pathThe full path of the root znode where ResourceManager state is stored. The default value is/rmstore.
yarn.resourcemanager.zk-num-retries

Number of times the ResourceManager tries to connect to the ZooKeeper server when the connection is lost.

The default value is 500.

yarn.resourcemanager.zk-retry-interval-msThe interval between retries, in milliseconds, when connecting to a ZooKeeper server. The default value is 2000.
yarn.resourcemanager.zk-timeout-ms

The ZooKeeper session timeout in milliseconds. The ZooKeeper server uses this configuration to determine session expiration.

Sessions expire when the server does not receive a heartbeat from the client within the session timeout period. The default value is 10000.

yarn.resourcemanager.zk-aclACLs that set permissions on ZooKeeper znodes. The default value is world:anyone:rwcda

 

 

  • No labels