This site contains release notes for MapR Version 5.0 and below.  You can also refer to the release notes for the latest release.

Skip to end of metadata
Go to start of metadata

What's New in Version 4.0.2

The 4.0.2 release of the MapR Distribution for Apache Hadoop contains the following new features. If you are upgrading from Version 3.1.x or earlier, see also the Release Notes for Version 4.0.1.

Disaster Recovery Improvement: Changing Mirrors into Read-Write Volumes

MIrror volumes can be promoted to standard read-write volumes, which is especially useful in disaster recovery scenarios. If a standard volume with critical data goes down in a primary datacenter, a mirror volume in a remote datacenter can be promoted to a standard volume in order to maintain business continuity. You can also promote mirror volumes to standard volumes for testing purposes while preserving the original data on the source volume.

Known Issues with Promoted Mirror Volumes

Read the following limitations carefully before using this feature:

  • Existing pre-4.0.2 mirror volumes cannot be changed to read-write; only new 4.0.2 volumes can be changed from standard to mirror and from mirror to standard. Customers who upgrade to 4.0.2 need to follow some specific manual steps in order to use this feature for volumes that were in place before the upgrade. See the 4.0.2 documentation for details.
  • Promotable mirrors do not work when a single source volume has multiple mirror volumes within the same cluster
  • When you use promotable mirrors, the mirror volumes on the destination cluster must be configured with the same names and mounting structure as the volumes on the primary site. Using different names will result in unpredictable behavior and is not supported.

    If volumes on the source cluster use hierarchical mount points (where volumes A, B, and C are mounted at /A, /A/B, and /A/B/C respectively), the same hierarchical mounting structure must be reproduced once the corresponding mirror volumes are promoted on the destination cluster.

    Icon

    Mirror volumes cannot be mounted under other mirror volumes. If volume B is mounted at /A/B in the source cluster, it will be unmounted in the destination cluster.

    After you convert a mirror volume to a standard (read-write) volume, you must remount the volume before you can write to it.

    For more details, see Using Promotable Mirrors

Enterprise Integration: MapR POSIX Client

The MapR POSIX Client allows a data source or a data consumer to have its own dedicated, highly performant access to MapR clusters. The data is compressed and optionally encrypted for secure access. This client works for existing applications that have not been written to the HDFS API, easing and expanding the number of applications that can take advantage of the cluster. This feature is extremely useful for HPC or other applications to access or write data into a Hadoop cluster without a single line of code change.

Industry-Leading Rolling Upgrades

Scripted rolling upgrades are supported from Version 4.0.1 to Version 4.0.2. The rolling upgrade utility has changed for this release. See the Upgrade Guide for details.

Update to Hadoop 2.5.1

When you install or upgrade to 4.0.2, the cluster runs on the 2.5.1 version of Hadoop Common. Hadoop Common includes the utilities that support MapR core components, MapReduce v2, and other Hadoop open source components that run in the cluster.

Simplified YARN HA: Zero-Configuration Failover for the Resource Manager

Zero configuration failover provides automatic failover of the Resource Manager process to another node in the cluster with no failover-specific configuration. In a new installation, zero configuration failover is the default configuration. You can also configure an existing cluster to use zero configuration failover for the ResourceManager. With zero configuration failover, the ResourceManager role is installed on two or more nodes but the ResourceManager process only runs on one node in the cluster. You also do not need to re-run configure.sh on each client node and cluster node after you add or remove the ResourceManager role on a cluster node.

Centralized Logging for MapReduce on YARN

MapR's Centralized Logging feature is now available for MapReduce on YARN applications. It provides a job-centric or application-centric view of all log files generated by a MapReduce program. Centralized Logging is disabled by default for MapReduce jobs using JobTracker/TaskTracker and MapReduce on YARN applications. This feature is not available for non-MapReduce applications.

Simplified Patch Installation

The Patch Installer automates the process of applying patches to the nodes in a cluster. Once you obtain a patch from Customer Support, you can run the mapr-setup script to apply the patch to every node in your cluster.

Impersonation via the HBase REST Gateway

You can access MapR-DB tables with user impersonation via the HBase REST Gateway service. The Rest Gateway supports impersonation on both secure and non-secure clusters. The gateway contains changes in both HBase and MapR core packages and works only with MapR Version 4.0.2 and HBase 0.98.7.

cldbguts Utility

The cldbguts utility prints information about active container reports, full container reports, registration requests, MapR-FS heartbeats, NFS server heartbeats, and containers.

Service and Crosscluster Tickets

The maprlogin utility supports the generation of service tickets, which provide a long-lived authentication mechanism for applications and services running outside the cluster. You can also create crosscluster tickets, which are required when you set up mirroring across secure clusters.

Reporting of Volume Access Time

The maprcli dump volumeinfo, maprcli volume info, and maprcli volume list commands now include the last time that an operation occurred on a volume. This property can be used to determine which volumes are accessed regularly and is updated every 6 hours.

Metrics Database Not Yet Supported for YARN Applications

You cannot use the Metrics Database to record activity for applications that run in YARN (MRv2). The database only supports MRv1 jobs.

MapR Interoperability Matrix

See the Interoperability Matrix page for detailed information about MapR server, JDK, client, and ecosystem compatibility.

Ecosystem Support

The Hadoop ecosystem components are hosted in a repository that is specific to Version 4.x: http://package.mapr.com/releases/ecosystem-4.x

To see a list of components supported in Version 4.0.2, see Ecosystem Support Matrix. For the latest ecosystem information, see Hadoop Component Release Notes.

Operational Changes

Note the following functional changes in Version 4.0.2.

LRU Cache Configuration

The default value of mfs.cache.lru.sizes in mfs.conf was changed in Version 4.0.2.

  • Version 4.0.1: inode:6:log:6:meta:10:dir:40:small:15

  • Version 4.0.2: inode:3:meta:6:small:27:dir:15:db:20:val:3

The cache values are expressed as percentages, which vary based on the expected size of the data the node is required to cache. The goal is to achieve a state in which most of the required data comes directly from the cache. You may need to tune the cache percentages based on your cluster configuration and the workload on specific nodes. Non-default allocations tend to work better for nodes that run only CLDB and nodes that do not have CLDB but do have a heavy MapR-DB workload. Note the following recommendations.

For CLDB-only nodes, increase the size of the cache for Dir LRU to 40%: change dir:15 to dir:40

A CLDB-only node is a fileserver node that hosts only the CLDB volume mapr.cldb.internal (no user volume data is hosted on the node). Dir LRU is used to host B-tree pages.

For non-CLDB nodes with no MapR-DB workload, optimize the cache to host as many file pages as possible. Change the value of the parameter to: inode:3:meta:6:small:27:dir:6

The remainder of the cache will be used to cache file data pages.

Note: You need to restart MFS in order for the change in mfs.conf to take effect.

MapR-FS User Permissions

Starting in Version 4.0.2, MapR-FS verifies that a user has execute access on a directory before the user can access the file or directory within it. Therefore, when a file or directory create operation results in the implicit creation of intermediate directories, those directories are created with write and execute permissions for the user. Previously, the intermediate directories were created with the same permissions as a newly created file or directory.

For example, a user submits a request to create file /a/b/c with permissions 400 (r--r--r--) and the directories a and b do not exist. In this case, the file create operation must create these intermediate subdirectories. Starting in Version 4.0.2, these intermediate directories would be created with permissions 744 (rwxr--r--). Prior to 4.0.2, these directories were created with permissions 400 (r--r--r--).

Service Memory Allocation

The maprcli service list command now lists the memory allocated to each service running on a node. A new memory allocation alarm is raised when the percentage of system memory required to run services on a node exceeds the set threshold.

Behavior of the -schedule option (volume create/modify)

The -schedule option in the maprcli volume create and volume modify commands has changed its behavior in Version 4.0.2. You may need to update your scripts accordingly.

  • The new -mirrorschedule option now has the meaning that -schedule used to have, when applied to a mirror volume.

  • The -schedule option refers to the ID of a schedule. If a schedule ID is provided, the volume will automatically create snapshots (standard volume) or sync with its source volume (mirror volume) on the specified schedule. Use the schedule list command to find the ID of the named schedule you wish to apply to the volume.

See the 4.0.2 mirroring documentation and maprcli command descriptions for more details.

Reduction in On-Disk Footprint for Containers

In Version 4.0.2 CLDB metadata for containers consumes less disk space. This feature is automatically enabled with a fresh install. After an upgrade to 4.0.2, issue the following command to reduce the space required on-disk for each container:

# maprcli config save -values {cldb.reduce.container.size:1}

Known Issues

You may encounter the following known issues after upgrading to Version 4.0.2.

Resource Manager Issues

14696/15100: When automatic or manual ResourceManager failover is enabled and a job is submitted with impersonation turned ON by a user without impersonation privileges, the job submission eventually times out instead of returning an appropriate error. This behavior does not affect standard ecosystem services such as HiveServer because they are configured to run as the mapr user (with impersonation allowed). However, this problem does affect non-ecosystem applications or services that attempt to submit jobs with impersonation turned ON. MapR recommends that customers add the user in question to the impersonation list so that the job can proceed.

14907: When several jobs are submitted and the ResourceManager is using the ZKRMStateStore for failover, the cluster may experience ZooKeeper timeouts and instability. MapR recommends that customers always use the FileSystemRMStateStore to support ResourceManager HA. See Configuring the ResourceManager State Store.

15864: When the ApplicationMaster does not run on the same node as the ResourceManager, the ApplicationMaster UI does not work. In this case, you cannot view the application status or application logs until the application completes.

Workaround: Add the ResourceManager Web UI address property to the client's yarn-site.xml using the following syntax:  

 <property>

   <name>yarn.resourcemanager.webapp.address</name>

   <value>${yarn.resourcemanager.hostname}:8088</value>

 </property>

Note: If the ResourceManager process fails over to another node, you need to update the value of this property to reflect the webapp address of the new ResourceManager process.

Installation and Configuration Issues

CentOS Version 6.3 and Earlier: MapR installations on Version 6.3 and earlier may fail because of an unresolved dependency on the redhat-lsb-core package.

This problem occurs when the CentOS repository points to vault.centos.org, rather than mirror.centos.org. If you encounter this problem, use one of the following workarounds:


15201: The Quick Installer installation logs print "Configuring Hive" and "Configuring Spark" messages even when these components were not configured.  

16216: When you run configure.sh with the -HS option on client nodes, the mapred-site.xml is re-generated and does not retain existing user settings. 

16155: In order to reconfigure a Mac client from secure mode to non-secure mode (or vice versa), you must follow these steps:

  1. Manually remove the entry for the current cluster from: /opt/mapr/conf/mapr-clusters.conf

  2. Run configure.sh.

16386: If you enable centralized logging on a cluster that was using YARN log aggregation prior to upgrading to version 4.0.2, you can no longer access previously aggregated MapReduce logs from the HistoryServer UI.

Workaround: Perform the following steps to view previously aggregated MapReduce logs from the History Server UI:

  • Use the yarn logs command to retrieve the logs for each MapReduce application. The output of this command contains stdout, stderr, syslog with specific delimiters.

  • Parse the output of yarn logs command to create syslog, stdout, stderr files using UNIX tools such as sed or awk.

  • Add the syslog, stdout, stderr files to the centrallized logging directory with the following directory hierarchy: 

    /var/mapr/local/<NodeManager node>/logs/yarn/userlogs/application_<applicationID>/container_<containerID>/

You will need to create the application and container directories and provide the user that submitted the application the proper permissions on the files and directories. For example, if usera submitted the application, usera should have the following permissions on the directories and log files:

  • Application directories:
    drwxr-s--- 5 usera mapr 4096 2015-01-07 11:32 /var/mapr/local/qa-node101.qa.lab/logs/yarn/userlogs/application_<id>

  • Container directories:
    drwxr-s---   - usera mapr 3 2015-01-07 11:32 /var/mapr/local/qa-node101.qa.lab/logs/yarn/userlogs/application_<id>/container_<id>

  • Log files:
    -rw-r-----   2 usera mapr 290 2015-01-07 11:32 /var/mapr/local/qa-node101.qa.lab/logs/yarn/userlogs/application_<id>/container_<id>/stderr

Note: After you complete the workaround, you will also be able to run maprcli job linklogs on these logs.

Resolved Issues

The following issues are resolved in Version 4.0.2.

Installation

15206: The configure.sh script was incorrectly writing a renice value into the limits.conf file, causing errors to be logged. This behavior no longer occurs in Version 4.0.2, but if the renice value already exists in the file, you must delete it manually.

15409: The Quick Installer now uses /proc/partitions to correctly detect disk devices.

Metrics Database

15251: A configuration change for the disk balancer in the MCS caused a reset of the Metrics Database password.

15385: Able to start hoststats after upgrading to v4.0.1.

MapR-DB

15497: An MFS core dump occurred when pre-split tablets were merged.

15505: The size of the tablet MRU cache for MapR-DB clients is now configurable by means of the new fs.mapr.tabletlru.size.kb parameter in the core-site.xml file.

15562: A column family with a non-zero ttl (time to live) value could not be cached.

15563, 15753: MapR-DB gets consumed excessive CPU on the MFS thread.

15858: HBase returns results when you list a specific directory. You no longer have to specify the full directory path to a table.

15871: Super users can now use the maprcli table edit command to fix invalid access control expressions that have accidentally been set for the adminaccessperm parameter for tables.

16411: Automatic throttling no longer occasionally blocks all other MapR-DB operations.

Volumes, Mirrors, and Schedules

6607: The inuse flag of a schedule now is set to 1 when a schedule is being used by a volume.

10410: Mirror snapshots are now deleted according to their expiration dates.

11218: Using the JobTracker user interface or the command hadoop queue -info <queue name> -showJobs to query whether a FairScheduler pool exists no longer causes a pool with the queried name to be created if the pool does not exist.

14997: Duplicate snapshots on replicas are handled correctly.

14862: Mirror resync across clusters can tolerate network errors.

15418: Dump creation on a large volume could not continue and reported session ID mismatch errors.

15435: The volume force unmount command works, even if the name container is deleted.

16396: Task distribution now works correctly with label-based scheduling.

YARN

14916: Now that the zero configuration failover option for the ResourceManager is available, job clients no longer have to query each ResourceManager to determine the active ResourceManager.

15731: When a cluster with Resource Manager High Availability configured was switched from being a non-secure cluster to a secure cluster, the standby RM failed to transition to the active Resource Manager.

16441: The Node Manager UI now displays correct values for all YARN parameters.

CLDB

14719: In Version 3.1.1, new JVM agent code caused a core dump on the CLDB nodes. In Version 4.0.2, this agent is disabled.

14535: Under-replicated containers raised alarms that did not clear in a reasonable amount of time.

15088: Containers that had been dropped or deleted from a node were stuck in the “resync” or "inactive" state.

Security

13203: Running configure.sh with the -P parameter to specify a Kerberos principal for the CLDB, and also specifying values for the -C and -Z parameters, now also updates the principal in the mapr-clusters.conf and mapr.login.conf files.

14452: The mapr user is recognized as having cluster admin privileges, and can execute any maprcli node services commands.

15284: Service tickets allow for arbitrary lifetimes.

15822: The MCS and CLDB web servers now support excluded cipher suites.

MapReduce

9375: The -list-blacklisted-trackers option of the hadoop job command now provides the reasons that TaskTrackers were blacklisted.

11283: The new parameter mapred.job.impact.blacklisting in the mapred-site.xml file lets you specify whether failures of a job should count toward the threshold for blacklisting the job’s TaskTracker.

14021: Warden is no longer killed occasionally when a JobTracker or TaskTracker is restarted through maprcli or MCS.

14172: The script hadoop-daemon.sh no longer accumulates temporary files in the Hadoop pids directory when tasks are terminated.

14840: A memory adjustment for TaskTracker ignored the number of map slots and reduce slots specified in mapred-site.xml.

15132: The warden.conf file recognizes the mfs.heapsize.maxpercent parameter.

15636: TaskTracker lock contention caused delays in task execution as well as task failures.

16261: JobTracker has improved performance while it handles a large number of processes.

MapR-FS

11103: Debug log messages from multiple FileClient threads now respect line boundaries.

14316: The message "parent is not a dir" prints as a debug message instead of an error message in mfs logs.

15137: Stopping MFS processes during a mirror operation resulted in an assertion error.

15151: A restore dump of a large file failed with resync abort messages.

15178: The default value for the LRU cache was changed. See LRU Cache Configuration, earlier in this page.

15444: During resynchronization, MapR-FS core dumps no longer occur due to caching of the entire block to memory.

15488: MapR-FS now supports hdfsOpenFile(O_RDWR). This C API opens a file in read/write mode.

15549: Disk I/O throttling is now automatically disabled when mfs.ssd.trim.enabled=1 is set in the mfs.conf file.

15748: When increments and gets were run in parallel, performance was degraded for the increments.

15757: fsck no longer fails due to disk errors unrelated to MapR.

15843: fsck no longer crashes due to trace messages in I/O threads.

15874: A new gfsck enhancement fixes metadata inconsistency in MapR tables.

 15880: fsck finds and repairs metadata inconsistency errors that can cause FileServer to stop unexpectedly.

15915: When container resyncs for network mirroring start back up after network interruptions, they continue from where they left off instead of completely restarting.

15922: When deleting a large volume of files from nodes, the CPU usage no longer hangs at 100 percent on the nodes.

15929: Network throttling for resync is now disabled when the resync of a container takes longer than thirty minutes and when only one valid copy of a container exists.

15995: MapR-FS now limits the number of readdir() RPCs that run in parallel to avoid high memory usage and FileServer restarts.

16220: A bucket flush operation resulted in a MapR-FS stack overflow.

16257: MapR-DB client applications no longer time out while trying to send RPCs to a MapR-FS process (“Unable to obtain binding for request” error).

16370: Attempted connections from a web application to MapR-FS no longer cause Tomcat to shut down.

NFS

15648: NFS status checks on RHEL 7 no longer fail due to calls on the incorrect version of the init script.

Alarms

8790: When the service mapr-zookeeper qstatus finds that a running ZooKeeper service is neither a leader nor a follower, qstatus issues an alarm that can be read with the maprcli alarm list command.

MapR Control System (MCS)

14409: The license manager window in the MCS shows available nodes and maximum allowed nodes for a particular license.

Ecosystem

14543: Impala memory parameters were set incorrectly, causing random crashes and node failures.

15031: A few tasks failed to complete for a Hive job with thousands of tasks.

15281: HBase shell creates tables when using COMPRESSION 'LZ4.’

15653: A mapr-asynchbase package was missing from the ecosystem repository for Ubuntu.

Build and Package

15286: The MapR distribution of Hadoop no longer contains commons-logging-1.0.4.jar, which is a third-party jar that is no longer required and caused memory leaks.

  • No labels