New in This Release
Services Down Alarm Removed
The Services Down Alarm (NODE_ALARM_MISC_DOWN) has been removed.
Hoststats Service Down Alarm Added
The Hoststats Service Down Alarm (NODE_ALARM_SERVICE_HOSTSTATS_DOWN) has been added. This alarm indicates that the Hoststats service on the indicated node is not running.
Installation Directory Full Alarm Added
The Installation Directory Full Alarm (NODE_ALARM_OPT_MAPR_FULL) has been added. This alarm indicates that the
/opt/mapr directory on the indicated node is approaching capacity.
Root Partition Full Alarm Added
The Root Partition Full Alarm (NODE_ALARM_ROOT_PARTITION_FULL) has been added. This alarm indicates that the
/ directory on the indicated node is approaching capacity.
Cores Present Alarm Added
The Cores Present Alarm (NODE_ALARM_CORE_PRESENT) has been added. This alarm indicates that the a service on the indicated node has crashed, leaving a core dump file.
Global fsck scan
Global fsck automatically scans the entire MapR cluster for errors. If an error is found, contact MapR Support for assistance.
A volume mirror is a full read-only copy of a volume that can be synced on a schedule to provide point-in-time recovery for critical data, or for higher-performance read concurrency. Creating a mirror requires the
mir permission. See Managing Volumes.
- (Issue 3724) Default Settings Must Be Changed
- (Issue 3620) Can't Run MapReduce Jobs as Non-Root User
- (Issue 2434) Mirroring Disabled in Alpha
- (Issue 2282) fsck Not Present in Alpha
The MapR Beta release may experience problems when nodes are removed from the cluster. The problems are likely to be seen as inconsistencies in the GUI and can be corrected by stopping and restarting the CLDB process. This behavior will be corrected in the GA release.
(Issue 4068) Upgrading Red Hat
When upgrading MapR packages on nodes that run Red Hat, you should only upgrade packages if they appear on the following list:
Other installed packages should not be upgraded. If you accidentally upgrade other packages, you can restore the node to proper operation by forcing a reinstall of the latest versions of the packages using the following steps:
- Log in as
sudofor the following steps).
- Stop the warden:
service mapr-warden stop
- If Zookeeper is installed and running, stop it:
service mapr-zookeeper stop
- Force reinstall of the packages by running
yum reinstallwith a list of packages to be installed. Example:
yum reinstall mapr-core mapr-zk-internal
- If ZooKeeper is installed on the node, start it:
service mapr-zookeeper start
- Start the warden:
service mapr-warden start
(Issue 4037) Starting Newly Added Services
After you install new services on a node, you can start them in two ways:
- Use the MapR Control System, the API, or the command-line interface to start the services individually
- Restart the warden to stop and start all services on the node
If you start the services individually, the node's memory will not be reconfigured to account for the newly installed services. This can cause memory paging, slowing or stopping the node. However, stopping and restarting the warden can take the node out of service.
For best results, choose a time when the cluster is not very busy if you need to install additional services on a node. If that is not possible, make sure to restart the warden as soon as it is practical to do so after installing new services.
(Issue 4024) Hadoop Copy Commands Do Not Handle Broken Symbolic Links
hadoop fs -copyToLocal and
hadoop fs -copyFromLocal commands attempt to resolve symbolic links in the source data set, to create physical copies of the files referred to by the links. If a broken symbolic link is encountered by either command, the copy operation fails at that point.
(Issue 4018)(HDFS-1768) fs -put crash that depends on source file name
Copying a file using the
hadoop fs command generates a warning or exception if a corresponding checksum file
.*.crc exists. If this error occurs, delete all local checksum files and try again. See http://firstname.lastname@example.org/msg15824.html
(Issue 3965) Volume Dump Restore Failure
volume dump restore command can fail with error 22 (
EINVAL) if nodes containing the volume dump are restarted during the restore operation. To fix the problem, run the command again after the nodes have restarted.
(Issue 3984) HBase Upgrade
If you are using HBase and upgrading during the MapR beta, please contact MapR Support for assistance.
(Issue 3890) Sqoop Requires HBase
The Sqoop package requires HBase, but the package dependency is not set. If you install Sqoop, you must also explicitly install HBase.
(Issue 3817) Increasing File Handle Limits Requires Restarting PAM Session Management
If you're upgrading from the Apache distribution of Hadoop on Ubuntu 10.x, it is not sufficient to modify
/etc/security/limits.conf to increase the file handle limits for all the new users. You must also modify your PAM configuration, by adding the following line to
/etc/pam.d/common-session and then restarting the services:
session required pam_limits.so
(Issue 3560) Intermittent Scheduled Mirror Failure
Under certain conditions, a scheduled mirror ends prematurely. To work around the issue, re-start mirroring manually. This issue will be corrected in a post-beta code release.
(Issue 3524) Apache Port 80 Open
The MapR UI runs on Apache. By default, installation does not close port 80 (even though the MapR Control System is available over HTTPS on port 8443). If this would present a security risk to your datacenter, you should close port 80 manually on any nodes running the MapR Control System.
(Issue 3488) Ubuntu IRQ Balancer Issue on Virtual Machines
In VM environments like EC2, VMWare, and Xen, when running Ubuntu 10.10, problems can occur due to an Ubuntu bug unless the IRQ balancer is turned off. On all nodes, edit the file
/etc/default/irqbalance and set
ENABLED=0 to turn off the IRQ balancer (requires reboot to take effect).
(Issue 3244) Volume Mirror Issue
If a volume dump restore command is interrupted before completion (killed by the user, node fails, etc.) then the volume remains in the "Mirroring in Progress" state. Before retrying the
volume dump restore operation, you must issue the
volume mirror stop command explicitly.
(Issue 3122) Mirroring with fsck-repaired volume
If a source or mirror volume is repaired with
fsck then the source and mirror volumes can go out of sync. It is necessary to perform a full mirror operation with
volume mirror start -full true to bring them back in sync. If a mirror operation is not feasible (due to bandwith constraints, for example), then you should restore the mirror volume from a full dump file. When creating a dump file from a volume that has been repaired with
fsck, use the
volume dump create command without specifying
-s to create a full volume dump.
(Issue 3028) Changing the Time on a ZooKeeper Node
To avoid cluster downtime, use the following steps to set the time on any node running ZooKeeper:
- Use the MapR Dashboard to check that all configured ZooKeeper services on the cluster are running. Start any non-running ZooKeeper instances.
- Stop ZooKeeper on the node:
service mapr-zookeeper stop
- Change the time on the node or sync the time to NTP.
- Start ZooKeeper on the node:
service mapr-zookeeper start
(Issue 2949) NFS Mounting Issue on Ubuntu
When mounting a cluster via NFS, you must include the
vers=3 option, which specifies NFS protocol version 3.
If no version is specified, NFS uses the highest version supported by the kernel and
mount command, which is most cases is version 4. Version
4 is not yet supported by MapR-FS NFS.
(Issue 2815) File Cleanup is Slow
After a MapReduce job is completed, cleanup of files and directories associated with the tasks can take a long time and tie up the TaskTracker node. If this happens on multiple nodes, it can cause a temporary cluster outage. If this happens, check the JobTracker View and make sure all TaskTrackers are back online before submitting additional jobs.
(Issue 2809) NFS Dependencies
If you are installing the MapR NFS service on a node that cannot connect to the standard apt-get or yum repositories, you should install the following packages by hand:
- Red Hat:
(Issue 2495) NTP Requirement
To keep all cluster nodes time-synchronized, MapR requires NTP to be configured and running on every node. If server clocks in the cluster drift out of sync, serious problems will occur with HBase and other MapR services. MapR raises a Time Skew alarm on any out-of-sync nodes. See http://www.ntp.org/ for more information about obtaining and installing NTP. In the event that a large adjustment must be made to the time on a particular node, you should stop ZooKeeper on the node, then adjust the time, then restart ZooKeeper.
When a node's IP number is reassigned to another node, remote procedure calls are misdirected to the wrong node. This can result in widespread errors affecting any system that use the RPC binding layer, including container resync operations that never finish or the creation of several superfluous containers. Restart the cluster to clear this condition.