8.4. Using stacked DRBD resources in Pacemaker clusters

Stacked resources allow DRBD to be used for multi-level redundancy in multiple-node clusters, or to establish off-site disaster recovery capability. This section describes how to configure DRBD and Pacemaker in such configurations.

8.4.1. Adding off-site disaster recovery to Pacemaker clusters

In this configuration scenario, we would deal with a two-node high availability cluster in one site, plus a separate node which would presumably be housed off-site. The third node acts as a disaster recovery node and is a standalone server. Consider the following illustration to describe the concept.

Figure 8.1. DRBD resource stacking in Pacemaker clusters

drbd-resource-stacking-pacemaker-3nodes

In this example, alice and bob form a two-node Pacemaker cluster, whereas charlie is an off-site node not managed by Pacemaker.

To create such a configuration, you would first configure and initialize DRBD resources as described in Section 6.18, “Creating a three-node setup”. Then, configure Pacemaker with the following CRM configuration:

primitive p_drbd_r0 ocf:linbit:drbd \
        params drbd_resource="r0"

primitive p_drbd_r0-U ocf:linbit:drbd \
        params drbd_resource="r0-U"

primitive p_ip_stacked ocf:heartbeat:IPaddr2 \
        params ip="192.168.42.1" nic="eth0"

ms ms_drbd_r0 p_drbd_r0 \
        meta master-max="1" master-node-max="1" \
        clone-max="2" clone-node-max="1" \
        notify="true" globally-unique="false"

ms ms_drbd_r0-U p_drbd_r0-U \
        meta master-max="1" clone-max="1" \
        clone-node-max="1" master-node-max="1" \
        notify="true" globally-unique="false"

colocation c_drbd_r0-U_on_drbd_r0 \
        inf: ms_drbd_r0-U ms_drbd_r0:Master

colocation c_drbd_r0-U_on_ip \
        inf: ms_drbd_r0-U p_ip_stacked

colocation c_ip_on_r0_master \
        inf: p_ip_stacked ms_drbd_r0:Master

order o_ip_before_r0-U \
        inf: p_ip_stacked ms_drbd_r0-U:start

order o_drbd_r0_before_r0-U \
        inf: ms_drbd_r0:promote ms_drbd_r0-U:start

Assuming you created this configuration in a temporary file named /tmp/crm.txt, you may import it into the live cluster configuration with the following command:

crm configure < /tmp/crm.txt

This configuration will ensure that the following actions occur in the correct order on the alice/ bob cluster:

  1. Pacemaker starts the DRBD resource r0 on both cluster nodes, and promotes one node to the Master (DRBD Primary) role.
  2. Pacemaker then starts the IP address 192.168.42.1, which the stacked resource is to use for replication to the third node. It does so on the node it has previously promoted to the Master role for r0 DRBD resource.
  3. On the node which now has the Primary role for r0 and also the replication IP address for r0-U, Pacemaker now starts the r0-U DRBD resource, which connects and replicates to the off-site node.
  4. Pacemaker then promotes the r0-U resource to the Primary role too, so it can be used by an application.

Thus, this Pacemaker configuration ensures that there is not only full data redundancy between cluster nodes, but also to the third, off-site node.

[Note]Note

This type of setup is usually deployed together with DRBD Proxy.

8.4.2. Using stacked resources to achieve 4-way redundancy in Pacemaker clusters

In this configuration, a total of three DRBD resources (two unstacked, one stacked) are used to achieve 4-way storage redundancy. This means that of a 4-node cluster, up to three nodes can fail while still providing service availability.

Consider the following illustration to explain the concept.

Figure 8.2. DRBD resource stacking in Pacemaker clusters

drbd-resource-stacking-pacemaker-4nodes

In this example, alice, bob, charlie, and daisy form two two-node Pacemaker clusters. alice and bob form the cluster named left and replicate data using a DRBD resource between them, while charlie and daisy do the same with a separate DRBD resource, in a cluster named right. A third, stacked DRBD resource connects the two clusters.

[Note]Note

Due to limitations in the Pacemaker cluster manager as of Pacemaker version 1.0.5, it is not possible to create this setup in a single four-node cluster without disabling CIB validation, which is an advanced process not recommended for general-purpose use. It is anticipated that this is being addressed in future Pacemaker releases.

To create such a configuration, you would first configure and initialize DRBD resources as described in Section 6.18, “Creating a three-node setup” (except that the remote half of the DRBD configuration is also stacked, not just the local cluster). Then, configure Pacemaker with the following CRM configuration, starting with the cluster left:

primitive p_drbd_left ocf:linbit:drbd \
        params drbd_resource="left"

primitive p_drbd_stacked ocf:linbit:drbd \
        params drbd_resource="stacked"

primitive p_ip_stacked_left ocf:heartbeat:IPaddr2 \
        params ip="10.9.9.100" nic="eth0"

ms ms_drbd_left p_drbd_left \
        meta master-max="1" master-node-max="1" \
        clone-max="2" clone-node-max="1" \
        notify="true"

ms ms_drbd_stacked p_drbd_stacked \
        meta master-max="1" clone-max="1" \
        clone-node-max="1" master-node-max="1" \
        notify="true" target-role="Master"

colocation c_ip_on_left_master \
        inf: p_ip_stacked_left ms_drbd_left:Master

colocation c_drbd_stacked_on_ip_left \
        inf: ms_drbd_stacked p_ip_stacked_left

order o_ip_before_stacked_left \
        inf: p_ip_stacked_left ms_drbd_stacked:start

order o_drbd_left_before_stacked_left \
        inf: ms_drbd_left:promote ms_drbd_stacked:start

Assuming you created this configuration in a temporary file named /tmp/crm.txt, you may import it into the live cluster configuration with the following command:

crm configure < /tmp/crm.txt

After adding this configuration to the CIB, Pacemaker will execute the following actions:

  1. Bring up the DRBD resource left replicating between alice and bob promoting the resource to the Master role on one of these nodes.
  2. Bring up the IP address 10.9.9.100 (on either alice or bob, depending on which of these holds the Master role for the resource left).
  3. Bring up the DRBD resource stacked on the same node that holds the just-configured IP address.
  4. Promote the stacked DRBD resource to the Primary role.

Now, proceed on the cluster right by creating the following configuration:

primitive p_drbd_right ocf:linbit:drbd \
        params drbd_resource="right"

primitive p_drbd_stacked ocf:linbit:drbd \
        params drbd_resource="stacked"

primitive p_ip_stacked_right ocf:heartbeat:IPaddr2 \
        params ip="10.9.10.101" nic="eth0"

ms ms_drbd_right p_drbd_right \
        meta master-max="1" master-node-max="1" \
        clone-max="2" clone-node-max="1" \
        notify="true"

ms ms_drbd_stacked p_drbd_stacked \
        meta master-max="1" clone-max="1" \
        clone-node-max="1" master-node-max="1" \
        notify="true" target-role="Slave"

colocation c_drbd_stacked_on_ip_right \
        inf: ms_drbd_stacked p_ip_stacked_right

colocation c_ip_on_right_master \
        inf: p_ip_stacked_right ms_drbd_right:Master

order o_ip_before_stacked_right \
        inf: p_ip_stacked_right ms_drbd_stacked:start

order o_drbd_right_before_stacked_right \
        inf: ms_drbd_right:promote ms_drbd_stacked:start

After adding this configuration to the CIB, Pacemaker will execute the following actions:

  1. Bring up the DRBD resource right replicating between charlie and daisy, promoting the resource to the Master role on one of these nodes.
  2. Bring up the IP address 10.9.10.101 (on either charlie or daisy, depending on which of these holds the Master role for the resource right).
  3. Bring up the DRBD resource stacked on the same node that holds the just-configured IP address.
  4. Leave the stacked DRBD resource in the Secondary role (due to target-role="Slave").