Red Hat Enterprise Virtualization based disaster recovery


The Problem

Low cost disaster recovery of virtualised server farms

Virtualised server farms always have shared storage. To have shared storage working in DR requires replication over WAN. The cost of low latency high speed WAN connections normally associated with DR solutions is prohibitively expensive for most SMEs and in some cases is not possible even for the most wealthy organisation. This therefore prevents them from establishing a meaningful, reliable DR solution.

The primary barrier is not bandwidth or cost but latency. Why is latency so important?

DR locations need to be sufficiently far apart as to enable some sensible amount of protection: 100m is too close. However to have a DR centre in an alternate city, potentially spanning tens of miles will always have some latency. Even if data travels at the speed of light (which would suggest minimal delay in getting from A to B) latency is introduced by all the devices that it travels through on the way.

Efficient usage of available resources

SMEs would like to minimise the capital requirements of establishing DR, either by purchasing the minimum amount of DR hardware or by over committing and existing alternate site.

The Solution

Without using storage level replication...

If there was storage level replication that would not only copy over data but would also move virtualised environment configuration

  • Replicate the virtual environment configuration using API's at both ends, reading from one and writing to the other, synchronise a subset of the configuration of the primary site onto an over committed subset of the secondary site.
  • Replicate the data using a method feasible on the type of WAN link that is available.

Our script runs on a preconfigured schedule (using cron in Linux) continuously synchronising the configuration on the primary site onto the secondary site. At the source, the API reads the configuration off the source RHEV-M server and saves it or saves it and then updates the configuration at the destination. If the configuration is only written, it is possible to manually import the saved configuration into the secondary site.

Periodic replication of the virtual machines.

Replication of the configuration of the machine whenever it changes. If I change the VM parameters in RHEV on the primary, the changes are written across to the secondary environment straight away via a script to the RHEV REST API.

The image of the VM is backed up and the back-up file shipped via rsync from the primary to the secondary. Scheduled incrmental back-ups are managed via Acronis and the incremental files are also shipped. Neither the full back-up, nor the incremental files are restored in the secondary environment.

RVSR setup

There are two ways of installing rvsr depending on if you want to install libraries and be compiling on the machine (needed for graphing the environment) or if you just want to install the minimal solution that will just run.

The "Just run it" option

ssh onto the server that will run as the rvsr and, with root permissions, make a directory in /opt

sudo mkdir -pv /opt/rvsr

Move into that directory and, with root permissions, copy down the code from github

curl > && sudo unzip -q && mv rvsr-master/* . && rmdir rvsr-master && rm -f

Edit the config.xml file (instructions below)

Run the rvsr tool


The "Full Python" option

Login as ROOT on your RHEV-M DR server

Makes sure python is installed

python --version

Create directory /opt/rvsr

sudo mkdir -pv /opt/rvsr

Python-pip will install the pip installer and it also is the only way to install easy_install

yum install python-pip

Virtualenv will be used to set up virtual python environments which will then protect your system against whatever you install within an environment

easy_install virtualenv

Create the virtual envonment and start it

cd /opt/rvsr/
virtualenv env
cd env
. bin/activate

Download the code from the repo. This should include requests and networkx. If it doesn't then you could consider using easy_install requests.

curl > && sudo unzip -q && mv rvsr-master/* . && rmdir rvsr-master && rm -f

If you want to have graphical representation of the environment then you need to have a graphviz environment set up.

yum install python-devel
sudo curl sudo curl -O > /etc/yum.repos.d/graphviz-rhel.repo
yum install gcc
yum install graphviz-devel
easy_install pygraphviz

Information: In a future edition the rvsr code will check to see if graphviz is installed before attempting to run it

Edit the config.xml file

Edit the config.xml file to reflect your environment

vim /opt/rvsr/config.xml

CONFIG.XML file as below for your reference

information USE 'https://IP.AD.DR.ESS:8443/api' for RHEV3.0 and 'https://IP.AD.DR.ESS/api' for RHEV3.1

The part of the XML file has a couple of simple tags that you can configure to suit your needs. Simple explanation of the tags are as follows:

  • refresh - Set TRUE if you want to 'Pull' settings from the Primary site
  • replicate - Set TRUE if you want to 'Push' setting from the Secondary site
  • verify_ssl - Set TRUE if you are using an SSL certificate to verify to the RHEV API





Setting up RVSR to run Daily

Add user rvsr to run the script

useradd --system rvsr

Create cron entry in cron.d (Daily) to run RVSR once a day

echo '0 * * * * rvsr /usr/bin/python /opt/rvsr/ 1>> /var/log/rvsr.log 2>> /var/log/rvsr.err' > /etc/cron.d/rvsr

Create log and error files for RVSR

touch /var/log/rvsr.log /var/log/rvsr.err

Change ownership of log and error files to rvsr user

chown rvsr:rsvr /var/log/rvsr.log /var/log/rvsr.err

Watch it in action

At Quru we're very excited about open source and are always working on unique ways to integrate different open source technologies into new solutions that make life easier and simpler.

Our QuruLab is a portable, mobile datacentre that we use to demonstrate the technology we are passionate about. We can bring it to your offices in a small case and have it running in a couple of minutes.

We have all sorts of interesting Red Hat, EnterpriseDB and Puppet Labs use cases running so if you'd like to see the above solution in action, or any of the other use cases we have developed, swing by the offices or we can come to you!

Call us on 0207 160 2883 or email [email protected]

About Quru

Quru is a market leader in the technical development, deployment and support of Linux and open source solutions that help organisations to reduce costs and increase operational agility and capability. We have also developed multiple award-winning software solutions ranging from mobile phone apps to global enterprise systems. Quru is based in Somerset House on the banks of the Thames, right in the centre of London. More...

Quru :: inspired open source solutions