In general, mirror volumes are created for the purpose of preventing or minimizing data loss. Data loss scenarios range from accidental overwrites to rack failure, to a disaster that destroys an entire datacenter. Mirror volumes are also used to improve performance or to make copies of data for use in other clusters without impacting production.
As of the 4.0.2 release, all new mirror volumes can be made into read-write volumes. In addition, read-write volumes that were mirrored to other volumes can be made into mirrors (to establish a mirroring relationship in the other direction). This functionality is useful in scenarios such as:
- Disaster recovery
If a read-write volume with critical data goes down in a primary datacenter, a mirror volume in a remote datacenter can be made into a read-write volume in order to maintain business continuity. Later, if the primary datacenter comes back online, the original mirror relationship can be restored by making the new read-write volume back into a mirror volume.
- Running applications on a copy of production data
- Resynchronization (reestablishing a mirror relationship after it is broken)
The remainder of this page focuses on how to incorporate mirror volumes into a disaster recovery plan. Of course, the techniques explained here can be used in other scenarios, too.
Incorporating Mirror Volumes into a Disaster Recovery Plan
Mirroring critical data to a remote datacenter (with the ability to make mirror volumes into read-write volumes) addresses these two objectives:
Recovery Point Objective (RPO) - the age of the files you need to recover (and how much data you can afford to lose)
Recovery Time Objective (RTO) - how soon you need to have a working datacenter in order to maintain business continuity
In a typical scenario that employs remote mirrors, the contents of a source volume are mirrored to a mirror volume in a remote cluster at a frequency specified by the mirror schedule. At the start of each mirror operation, a snapshot is taken of the source volume's contents. The mirror operation takes some time to complete, and while the data is being copied from the snapshot to the mirror volume, more data is written to the source volume. This data will be captured during the next mirror operation. When each mirror operation completes, the contents of the mirror volume are identical to the contents of the source volume at the time of the snapshot. For subsequent mirror operations, only the incremental changes (additions and deletions) are copied to the mirror volume, which synchronizes its contents with the contents of the source volume at the time of the snapshot.
If the source cluster goes down, any data written to the source volume since the last successful mirror operation cannot be copied to the mirror. The amount of data lost depends on the number of write operations in the interval from the last successful mirror to the time the cluster goes down.
Factors that Affect RTO
During a disaster, an administrator must first determine that the link is down between the primary datacenter and the secondary datacenter. Next, the administrator begins the process of switching applications that were running on the primary datacenter over to the secondary datacenter. For write applications, the administrator begins converting mirror volumes to read-write volumes, starting with volumes that contain the most critical data. Note that read applications can run on read-only mirrors, but write applications can only run on read-write volumes.
To gauge how long it will take to switch applications from the primary datacenter to the secondary datacenter (and to set the RTO accordingly), consider these factors:
- Detection time (how long it takes to determine that the link is down between the two datacenters)
- Switching time (how long it takes to switch applications from one datacenter to the other)
- Promotion time (how long it takes to change read-only mirror volumes to read-write volumes that can run write applications). Promotion time is a function of:
- the size of each volume (larger volumes take longer to change than smaller ones)
- the number of files in each volume (the smaller the number of files to change, the faster the conversion)
- the size of each file (file size determines number of files - the larger each file is, the fewer files you need to store data; fewer files take less time to change)
- the number of containers in each volume
- the number of volumes
- Whether mirror throttling is enabled (the default) or disabled (which speeds up the mirroring process)
Factors that Affect RPO
Various factors affect how much data can be recovered through the use of mirror volumes. To specify a realistic recovery point objective in your disaster recovery plan, take these factors into account:
- Mirror schedule (how often the mirror is synchronized with its source volume)
Note that the first mirror operation is a full synchronization between source and mirror volumes. Subsequent mirror operations are incremental - only the changes that occurred since the last mirror event need to be copied in order to synchronize the contents between the two volumes.
- Network link between the source volume and the mirror volume (consider the stability and quality of the link, as well as latency, throughput, and other activities across the link)
Using Promotable Mirrors for Disaster Recovery
The following sections describe the tasks that a MapR administrator performs from a remote datacenter before, during, and after a disaster. These tasks include:
For a brief overview of the terminology used to describe volume types, along with some basic commands, see the Glossary.
Setting up Mirroring to a Remote Cluster
Once data volumes are created in a primary datacenter, the MapR administrator creates mirror volumes in a remote secondary datacenter. The following diagram illustrates the mirror relationship between these two volumes:
Before you can create a mirror volume on a remote cluster, you must first complete these steps:
- Check UID consistency and volume permissions
- Edit mapr-clusters.conf so every node in each cluster can resolve all nodes in the other cluster.
- (For mirroring between secure clusters only) Generate a cross-cluster ticket and append it to the
maprserverticketon the destination cluster.
Instructions for each step are explained below.
Checking UIDs and Setting Volume Permissions
Make sure you have the same UID for the
MAPR_USER (the cluster owner) for both the primary cluster (where the source volume resides) and the remote clusters (where the mirror volumes reside; also known as the destination clusters). You also need to have these volume permissions:
dumppermission on the source volumes
restorepermission on the mirror volumes at the destination clusters
Another requirement for mirroring to work between two clusters is that every node in each cluster (the local cluster and the remote cluster) must be able to resolve all nodes in the other cluster. So, before you create a mirror volume, edit the
mapr-clusters.conf file on each node as shown:
For each node on the source volume's cluster, add a line that contains the mirror cluster's name, followed by a space-separated list of hostnames for its CLDB nodes. For example, if Cluster1 has cldb1-1 and cldb1-2 for its CLDB nodes, and Cluster 2 has cldb2-1 and cldb2-2 for its CLDB nodes, the
mapr-clusters.conffile for Cluster1 would have these lines:
For each node on the mirror volume's cluster, add a line that contains the source cluster's name, followed by a space-separated list of hostnames for its CLDB nodes. For this example, the
mapr-clusters.conffile for Cluster2 contains these lines:
secure=trueif both clusters are secure. Set
secure=falseif both clusters are not secure.
On each cluster, restart the
mapr-webserverservice on all nodes where it is running.
Generating a Cross-cluster Ticket
If the source volume at the primary datacenter is in a secure cluster, the destination cluster needs authorization to pull data from the source cluster in order to create a mirror volume. Authorization is granted by means of a cross-cluster ticket generated by the source cluster administrator. Each step in the process is explained below:
(Source cluster administrator) Define a new service user in the source cluster’s UNIX user registry. One way to do this is with the
useraddcommand (as appropriate for your operating system). For example:
(Source cluster administrator) Generate a cross-cluster ticket for the user you created in step 1. This command can be run from any node in the source cluster.
The input ticket file specified by
-inmaprserverticketfileis the source cluster’s server ticket, which is located at
/opt/mapr/conf/maprserverticket. The output ticket file, specified by
ticketfile, contains the cross-cluster ticket.
(Source cluster administrator) Provide the cross-cluster ticket to the destination cluster administrator.
(Destination cluster administrator) Append the cross-cluster ticket file (located at
/opt/mapr/conf/maprclusterticket) to the
/opt/mapr/conf/maprserverticket) on the destination cluster.
- (Destination cluster administrator) Copy the ticket file to all nodes in the destination cluster.
Creating a Mirror from the Command Line
To set up a mirror volume in a remote cluster at the secondary datacenter, run the
volume create command, as shown in this example:
When you run this command, a mirror volume named
volA is created with these specifications:
|Option||Description||Value used in example|
|Name of the mirror volume|
|Path to the mirror volume (the mount point)|
|Type of volume (either |
|Name of the source volume from which the mirror pulls data. For remote mirroring, the name must include the cluster name, since the source is located on a different cluster from the mirror.|
|The ID corresponding to the snapshot schedule for the mirror|
volume. These schedule IDs are pre-assigned:
See Default and User-defined Schedules for information on creating a custom schedule.
|The ID corresponding to the mirror schedule. These schedule IDs are pre-assigned:||1 (for critical data)|
Creating a Mirror from the MCS
To create a mirror from the MCS, follow these steps:
- Under the MapR-FS group in the navigation pane, click on Volumes, then click on the New Volume tab.
The New Volume dialog box displays.
- Fill out the New Volume dialog box.
- Choose Remote Mirror Volume.
- Give the volume a name (the mirror is named volA in this example).
- Provide the name of the source volume and the cluster where it resides (the source volume is volA and the cluster is Cluster1 in this example).
- Supply the mount path (/A in this example).
Select a snapshot schedule (specifies when to take snapshots of the volume). The default choices are None, Critical data, Important data, and Normal data.
- Select a mirror schedule (specifies the mirroring interval of the volume). The default choices are None, Critical data, Important data, and Normal data. If you choose None, start the mirror manually by selecting Start Mirroring from the Volume Actions tab.
Choosing a Snapshot Schedule and a Mirror Schedule
When you specify a snapshot schedule on a mirror volume, it specifies how often to take a snapshot of the mirror volume. This snapshot schedule is distinct from the snapshot schedule for the source volume. A snapshot schedule for a promotable mirror volume has two purposes:
- It specifies how often to take a snapshot of the mirror volume for the purpose of preserving the state of the mirror before a subsequent mirror operation. This way, if corrupt data is copied from the source volume's snapshot into the mirror volume, the mirror contents can be rolled back to the snapshot.
- If the promotable mirror volume is promoted to a read-write volume, the snapshot schedule specified for the mirror is used for the promoted read-write volume. Once a mirror volume is promoted to a read-write volume, the mirror schedule is disabled.
A mirror schedule specifies how frequently the mirror volume is synchronized with the source volume. In case of a disaster (or any type of data loss on a read-write source volume), the data can be recovered from the mirror volume, but any data written to the source volume since the last successful mirror operation will not be on the mirror volume. Therefore, you should set the mirror schedule such that it meets your RPO (Recovery Point Objective).
Default and User-defined Mirror Schedules
When you create a mirror from the MCS, you can select from three classes of data that correspond to different default mirror schedules:
In addition, you can create your own user-defined schedule, which will also appear as a selection in the dropdown menu.
To define a new schedule:
- Select MapR-FS > Schedules from the MCS. Click the button. The following dialog appears.
- Enter a schedule name and schedule rules. Click to add more rules for mirror schedules. For example, a schedule named Customer Data could be defined like this:
- Click the button.
- Note that Customer Data now appears in the list of Schedule Names on the Schedules tab of the MCS.
- Note that when you create a new volume, the snapshot scheduling dropdown menu includes the new schedule.
Guidelines for Setting Mirror Schedules
Although MapR allows mirroring frequencies up to once per minute, setting a schedule at this frequency is not practical nor advisable. When you choose the mirror schedule, consider the amount of data on the volume and the load on the cluster. Remember that the mirroring frequency must allow enough time to complete one mirror operation before the next scheduled mirror operation starts. In addition, if you have a cascaded mirror setup (where A mirrors to B which mirrors to C), you cannot set a mirror schedule for C that starts before B finishes mirroring from A.
If you set a mirror schedule to start mirroring before the previous mirror operation finishes, you will see an error message similar to this:
Failing Over to a Mirror Volume
When a disaster occurs at a primary datacenter, data can no longer be written to the volumes in that location, and the mirror operation cannot be performed. In order to maintain business continuity, the administrator at the secondary datacenter promotes the read-only mirror volume to a read-write volume, which breaks the mirror relationship. At this point, the promoted mirror volume contains all the data that was on the source volume at the time of the most recent successful mirror operation.
Promoting a Volume from the Command Line
To promote a read-only mirror to a read-write volume from the command line, run the
volume modify command from the cluster where the mirror resides and specify the name of the mirror volume that is being promoted. In this example, the mirror volume is named volA:
To promote several mirror volumes at once, provide a comma-separated list of volume names. For example:
Promoting a Volume from the MCS
To promote a read-only mirror to a read-write volume from the MCS, follow these steps:
- Click on Mirror Volumes in the navigation pane, then check the box to the left of the volume you want to promote. You can promote more than one mirror at at time by checking multiple boxes.
- Click on the Volume Actions tab, then select Make Standard Volume from the dropdown menu.
Handling Mount Points in Promoted Mirror Volumes
After you promote read-only mirror volumes to read-write standard volumes, you must re-establish the mount points that were set up in the source cluster. To understand the steps in this process, consider the following scenario:
A source cluster has volumes A, B, and C, which are mounted at /A, /A/B, and /A/B/C respectively. Each source volume is mirrored to a volume in another cluster (the destination cluster). The names of the corresponding mirror volumes are also A, B, and C.
Mirror volume A is mounted at /A, but since the mirror is read-only, no mount point can be created beneath it for mirror B or mirror C.
Now suppose that all three mirror volumes are promoted to read-write volumes. Before any data can be written to these volumes, the volume links must be removed and the volumes must be remounted. The commands for each step are shown below.
Promote A, B, and C to read-write volumes.
Remove the vol links located at /A/B and /A/B/C. Since mirror A was already mounted, its vol links do not need to be removed.
Mount the promoted read-write volumes B and C at the same mount points used in the primary (source) cluster, in order to maintain an exact replica in the destination cluster.
Now the promoted volumes are accessible for write operations.
Changing the Limit for Concurrent Mirror Operations
The system allows a maximum of 50 concurrent mirroring operations by default. Mirroring operations include both mirroring and promoting from read-only mirrors to read-write standard volumes. The system parameter that controls this limit is
For large-scale mirror operations involving many volumes, a script automates the process. For example, if a script queues 100 volumes for mirroring operations, and the
mapr.mirror.concurrent.ops limit is set to 50, the mirroring operations start on the first 50 volumes in the queue. As soon as one volume completes, another volume is processed from the queue until all 100 are completed. Since volumes are processed from the queue in first-in first-out (FIFO) order, the script should specify the most critical volumes first.
If you want to process more volumes at a time, you can raise the limit of the
mapr.mirror.concurrent.ops parameter. To tune this parameter for maximum efficiency, consider the number of containers per volume. A higher number of containers per volume requires a lower limit than a lower number of containers per volume. To raise the limit to 500 for example, run the following command:
Restoring the Mirror Relationship
If the primary datacenter comes back online, the administrator can re-establish the mirror relationship between the original read-write volume in the primary datacenter and the promoted read-write volume in the secondary datacenter.
Note that the two read-write volumes will have different data, since data was written to the promoted mirror while the original source volume was down. The original source volume might also have different data that was written after the last mirror operation, but before the cluster went down. The administrator must decide which data to keep and use as the source.
Preserving volA/Cluster1's Data
Suppose that volA in the primary datacenter contains crucial data that must be preserved, and you want to mirror its data to volA in the secondary datacenter (the same mirror relationship that was established originally). To recreate the original mirror relationship, convert the promoted volume, volA/Cluster2, from a read-write volume to a mirror of volA/Cluster1 by running the following command:
To use the MCS to convert volA/Cluster2 from a read-write volume to a mirror of volA/Cluster1, perform these steps from Cluster2:
- Select MapR-FS > Volumes from the navigation pane and click on the checkbox next to the read-write volume you want to convert (volA in the example).
- Click on the Volume Actions tab and select Make Mirror Volume.
- Select MapR-FS > Mirror Volumes and verify that volA on Cluster2 is displayed as a Standard Mirror.
Preserving volA/Cluster2's Data on volA/Cluster1
Now suppose you want to preserve the data on volA/Cluster2 (in the remote datacenter) but you still want volA/Cluster1 to be the primary volume with volA/Cluster2 as its mirror. To save volA/Cluster2's data on volA/Cluster1 and reestablish the original mirror relationship from volA/Cluster1 to volA/Cluster2, follow the steps below.
From the command line
Stop writing new data to volA/Cluster2. To be sure no data is written to this volume, make it read-only by running this command:
Pull the data from volA/Cluster2 to volA/Cluster1 by making volA/Cluster1 a mirror of volA/Cluster2.
Start the mirror operation.
Once mirroring is done, promote volA to a read-write volume. Note that the mirror relationship breaks at this point.
Make volA/Cluster2 a mirror of volA/Cluster1.
From the MCS
- Stop writing new data to volA/Cluster2 by making this volume read-only:
- Click on the checkbox next to volA in the Volumes display.
- Click on the name of the volume to display the Volume Properties dialog.
- In the Volume Properties dialog, check the Read-only box and click OK.
- Make volA/Cluster1 a mirror of volA/Cluster2.
- Navigate to the node that is running the
mapr-webserverservice on Cluster1 to access the MCS.
- Select MapR-FS > Volumes from the navigation pane and click on the checkbox next to volA.
- From the Volume Actions tab, select Make Mirror Volume.
- Fill in the Source Volume name field (the source volume is volA in this example) and click OK.
- Navigate to the node that is running the
- Start mirroring.
- Promote volA to a read-write volume.
- Make volA/Cluster2 a mirror of volA/Cluster1.
Working with Volumes after Upgrading from 3.x or 4.0.1 to 4.0.2
If you created volumes in a cluster running 3.x or 4.0.1, and then you upgrade your cluster to 4.0.2, all the old volumes are preserved. When you create new volumes, the type (standard or non-convertible) is determined as follows:
- If you create a new read-write volume, it is a standard volume (non-convertible standard volumes cannot be created on a 4.0.2 cluster).
- If you create a mirror from a non-convertible standard volume, the mirror is a non-convertible mirror.
- If you create a mirror from a standard volume, the mirror is a standard mirror.
After you upgrade to 4.0.2, follow these steps so you can start using the promotable mirror volume functionality:
Enable the promotable mirror volume functionality by setting the parameter
- If a non-convertible standard volume contains critical data, move the data to a standard volume:
Create a new standard volume.
Stop writing to the non-convertible standard volume.
Copy the data from the existing standard volume into the convertible rw volume.
Create a promotable mirror volume on a remote cluster that can be used in case of a disaster.
If you use automated scripts that specify the old volume types 0 and 1, these types are mapped to
mirror respectively for backward compatibility. For example:
If a command uses
-type 0to create a standard rw volume, it is mapped to
-type rw. The resulting volume is a convertible rw volume (
is mapped to
If a command uses
-type 1to create a mirror volume from a non-convertible standard (read-write) volume (created before 4.0.2), it is mapped to
-type mirror. The resulting volume is a non-convertible mirror volume, since the source is a non-convertible standard volume.
is mapped to
If a command uses
-type 1to create a mirror volume from a standard rw volume, it is mapped to
-type mirror. The resulting volume is a standard mirror volume, since the source is a standard rw volume.
is mapped to
This glossary explains the terms used in the MCS for the different types of volumes. A sample display from the Type column is shown here:
|NC Standard Volume|
A non-convertible standard volume with read-write capabilities, created before MapR version 4.0.2. These volumes cannot be converted to standard mirror volumes. If this volume type is designated as a source volume when a mirror volume is created, the mirror volume will be an NC mirror volume.
When you move the mouse over the NC Standard Volume text in the MCS, the following tooltip is displayed:
An NC standard volume is designated as type 0 in the output of the
|Standard Volume||A read-write volume created with MapR version 4.0.2. A standard volume can be converted from read-write to mirror (read-only). If a mirror is created from this type of volume, the mirror can be promoted to a read-write volume.|
A standard volume is designated as type
|NC Mirror Volume|
A non-convertible read-only mirror volume created before MapR version 4.0.2. This volume type cannot be promoted to a read-write volume, and can only be created from an NC standard volume.
When you move the mouse over the NC Mirror Volume text in the MCS, the following tooltip is displayed:
An NC mirror volume is designated as type 1 in the output of the
A mirror volume that starts as read-only, and can be promoted to a read-write volume.
A standard mirror volume is designated as type