Stacy's Tech Blog

Neverfail is a wonderful business continuity product. I’ve been deploying it for a few years now, primarily for Microsoft SQL and Microsoft Exchange protection, and earlier this year, I had the opportunity to deploy Neverfail for Exchange to a virtualized environment.

My client has used Neverfail for the last couple of years with physical server pairs and crossover cabling for the Neverfail Channel, which is the traditional way to deploy this solution. So I deployed Neverfail for SQL 2005 and Neverfail for Exchange 2003 for them on separate projects a couple of years ago and away we happily went.

The main objective behind a virtualization intitiative, was to lay the foundation to allow for fast growth by leveraging virtualization technologies to conserve server resources and provision those resources quickly. We began planning a series of core infrastructure hardware upgrades to put them in a position to deliver these resources to the existing campuses and to new campuses as they came online. In addition, a planned upgrade (including virtualization) of the CampusVue architecture, the main Line of Business application, was in the planning stages and laying the groundwork now would be critical to ensuring success for that project. (I will post a Tech Note on CampusVue 11, n-tier virtualization deployment using XenServer in the near future).

Initially, I was concerned that we were going to have to select a different hypervisor and given the scope of the overall virtualization effort, was not really keen on that. But Citrix XenApp just plain outperforms the competition on XenServer and since the main Line of Business application for this client is delivered using Citrix Presentation Server 4.5, the choice was to go with XenServer 5.

Some months prior to planning these infrastructure upgrade initiatives it became apparent that there was one server in the Neverfail for Exchange server pair that was past end-of-life and the client was preparing to buy a new server. In fact, the Primary Server failed completely and was shut down for several months. I suggested that given the virtualization efforts, why not simply install the Neverfail Secondary server into a VM? In theory, it should work, but I began researching this as a viable option, since there was already a budget for virtualization server hardware, storage, core switching, etc.

Neverfail has quite an extensive knowledge base and I found several articles and spoke with several folks over there about this solution, but there just wasn’t much data. They have information in their KB regarding VMware but not much was published on XenServer. But since the product could work in ESX, it was a pretty safe bet that it would work on XenServer, so I began planning this during the initial virtualization deployment.

The initial deployment was a 2-node XenServer 5 resource pool using a Lefthand Networks iSCSI SATA SAN (pre-HP) that I had deployed during the initial Neverfail implementation. Originally that SAN had been used as a backup target and subsequently Exchange databases were moved to one of those LUNs (no, I didn’t do that part). I surmised that as long as we didn’t push any serious I/O through those LUNs, then we should be fine. In addition, I put in a temporary HP Gigabit switch to ensure that his throughput was a bit more reliable. These measures were temporary since we were expecting new hardware specifically ordered for this purpose.

After the nodes were deployed, I created the VM for the Neverfail for Exchange Secondary and began the re-deployment of Neverfail. I won’t go into the gory details from the Neverfail procedure (and it is quite detailed and wonderful), but I will abbreviate it here:

Build Secondary Server

1. Must have the same OS version, Service Packs & hotfixes
2. Must have same drive letter configuration with volume sizes => Primary Server
3. Must have same network configuration for LAN deployment

Deploy Neverfail software to Primary server. This process does the following:

1. Installs Neverfail software
2. Clones Primary Server using NT Backup
3. Reboot and server is again ready to provide services to users

Deploy Neverfail to Secondary server. This process does the following:

1. Installs Neverfail software
2. Restores “clone” to Secondary
3. Configure network appropriately
4. Reboot and servers will synch (a lengthy process)

This procedure is incredibly well documented by Neverfail. The only changes that I had to make on the fly are explained below. In addition, the XenServer Tools (PV drivers) MUST be installed prior to beginning ANY work on the Secondary. The IP configurations get all hosed if you don’t.

During the course of this implementation, I created VLANs on the client network, creating one for the Neverfail Channel. Since the XenServer NIC bonds were trunked to the switch, I had to do this to ensure that there was protected layer 2 communication between the physical Primary Server and the virtual Secondary Server network interfaces. Plus, it’s a standard practice when deploying XenServer .

The network portion of the XenServer resource pool, by default, is essentially a shared switch configuration. So, if viewed from that perspective, it’s quite easy to configure “virtual networks” with assigned VLANs for tagging traffic out the trunked NIC bonds (I used bonds in this scenario for most traffic, but a single NIC for the Neverfail Channel).

The “gotcha” comes during the portion of the Neverfail installation that asks you to ” disconnect the network cables from all network adapters on the Secondary server (the cables Must be disconnected rather than disabling the network adapters to proceed)”. I worked around this by disconnecting the Primary Server NICs, since that cannot be done in a XenServer VM. Since the restore process will actually overwrite IP configurations, this is critical to ensure that two Active servers are connected and visible to the network simultaneously, which could cause data loss and/or corruption.

During a Neverfail deployment, the Neverfail Primary server down-time window is equal to the time to install the Neverfail software + time to clone + one reboot. In a deployment to a virtual machine, this time is extended to include deployment of the Neverfail Secondary server to ensure there is no data loss. I am sure there are alternate ways to achieve the same result, but this is the way that I chose to ensure the integrity of the environment. If you have an alternate solution, then please chime in!

In any case, it’s quite a nifty little setup . When the remaining hardware was shipped and I built out the remaining components: 7 XenServer nodes, HP LeftHand P4500 and a Cisco 3750g stack (two nodes) and these components were deployed, we switched over to the Secondary (virtual) server for testing and it’s been running “virtually” ever since. Surprisingly enough, we were able to correct a memory problem by virtualizing this without service interruption. (More on Free Memory PTEs and Neverfail at a later time)

So all-in-all, Neverfail on XenServer is quite a nice leverage of virtualization technology and equals happiness for my client.