From Greg Porter's Wiki

Revision as of 22:51, 7 December 2009 by Greg (Talk | contribs)
(diff) ← Older revision | Current revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Openfiler and XenServer - Use Openfiler based iSCSI storage for XenServer

Shared storage? Why do I need that? Local drives work fine.

Well, when you're first experimenting with virtualization like XenServer, you might say that to yourself. The "why" soon becomes apparent, though, if you read the documentation (From the admin guide).

A resource pool comprises multiple XenServer host installations, bound together into a single managed
entity which can host Virtual Machines. When combined with shared storage, a resource pool enables VMs
to be started on any XenServer host which has sufficient memory and then dynamically moved between
XenServer hosts while running with minimal downtime (XenMotion).

That's the deal. Shared storage is the "secret sauce" that lets you move *RUNNING* machines between physical hosts with no downtime.

In my opinion, this is a requirement for production. You need a pool. You need at least 2 XenServer hosts in that pool. You need shared storage to store the virtual machines on. That way if a physical host needs maintenance, or need a software upgrade you:

  • Migrate all virtual machines (while running, no downtime) to the other host with XenMotion
  • Do the maintenance or upgrade
  • Move the machines to the newly repaired or updated host
  • Fix the remaining host
  • Spread the machines back out among the newly fixed or updated hosts

In a pool, with shared storage, you can do all that with no downtime for the virtual machines running your applications.

You could possible use NFS for shared storage. NFS shares work fine for storage for virtual machines. They are a supported type of storage repository. NFS is easy and cheap. Why not use it?

Because if you ever want to move later to implement the full version of XenServer (Citrix Essentials for Xenserver), iSCSI or Fibre Channel shared storage is *REQUIRED* - again, from the admin guide:

To use the HA feature, you need:
• Shared storage, including at least one iSCSI or Fibre Channel LUN of size 356MiB 
or greater -- the heartbeat SR. 
• A XenServer pool (this feature provides high availability at the server level 
within a single resource pool).

If you've never looked at shared storage you may be tempted to think you'll just go "downtown" and buy a iSCSI array. Sure, there's lots of vendors that'll sell you one. For a *LOT* of money. "Real" arrays are expensive. Something on the order of $20,000 for an entry level and up from there. Storage vendors also get you on upgrades. For example, Equallogic charges $2500 for a 1TB SATA drive that you can buy bare for less than $100. Of course, it does have a nice Equallogic mounting bracket on it, solid metal, no cheap plastic mounting brackets on Equallogic drives.

Using Openfiler iSCSI with XenServer

In my opinion, anyone using XenServer should be working on trying to get a "production like" environment up, in order to get the full experience. "Production like" in this case means making a resource pool, with two or more hosts in it, with shared storage. Even if you are short of the hardware to implement 2 hosts now, you can still make a "pool of one" and practice some pool operations.

So in the examples that follow, I'll use a simple pool with 2 hosts. If an operation is possible to do with XenCenter, I'll use it (and most operations are possible). So the examples will be using the GUI. Note that all XenServer operations are possible at the command line, as well.

Adding iSCSI shared storage to a XenServer pool

Assuming you have already set up an iSCSI target for XenServer use, it's really simple. Right click on the pool to add the storage to, and select New Storage Repository (SR).

New SR

Select iSCSI as the virtual disk storage type. (With "plain" iSCSI, you're not using StorageLink technology, this is special storage integration that comes with particular models of filers.)

Select iSCSI

For "Name" you can type a meaningful name for this iSCSI SR. This will be the name displayed in XenCenter for this SR. Type the iSCSI filer name in the "Target Host" field. (Note that if you have DNS issues, the IP address works fine as well.)

Enter SR name and iSCSI host name

Discover IQNs

Click on the "Discover IQNs" button. Xen will query the supplied host on port 3260, and request a list of targets available. This might take a moment.


If you are following my example, you have a very simple set of targets and LUNs. You have one target, with only one LUN mapped to it.

You should see the name of the target from the Openfiler server. The name you see in XenCenter will match the IQN in the Openfiler GUI.

If you don't get a IQN back, then go and recheck your Openfiler setup. Is the iscsi target service running? Is this host set to "allow" in the network access list? Did you reboot Openfiler just now?

Discover LUNs

Now hit the "Discover LUNs" button. You should get back the name of the LUN from Openfiler, and the names should match.

Discover LUNs

Make sure that the LUN displayed is the one you want to use. (You could have multiple LUNs behind one iSCSI target. If you are following my simple examples, you only have one).

Format the SR?

Add the SR

Adding the SR is a destructive operation. Xen will now format the new SR. Make very sure that this is the correct LUN and that it is OK to format it. If so, select yes.


That's it! You now have shared storage.


Note that with a pool, all physical hosts in the pool automatically get access to the shared storage. The first SR in the pool is automatically marked as default.

Using the shared storage

In the simple case, build a virtual machine in your pool. Since the iSCSI shared storage you just added is the default storage location, the vm will be built on the Openfilr provided iSCSI storage LUN.

In my case, I saved a copy of my Windows vm before I did the iSCSI SR operations. As soon as I got my iSCSI SR added, I imported my vm to the pool.

Virtual machine import

The vm "lives" in the iSCSI SR you just made with Openfiler.

Virtual machine on Openfiler

So why go through all that hassle? XenMotion, that's why. You can move the machine from host to host while running. Try it now, right click on the machine, Select Migrate, select the other server.


It'll take a few seconds. Your clue that it is happening is in the lower left corner where status messages appear. It will say "Migrating". After a few seconds, the machine will change host locations. For extra fun, try pinging it during a move. You might see one dropped ping packet. It happens so quick, users won't notice.

XenMotion completed

Troubleshooting and interesting usage notes

XenCenter is sort of oblivious as to the status of the iSCSI SRs it is using. In other words, say the Openfiler providing the backend for the SRs fails. You'd think you'd get some sort of unhappy "red x" or scary error messages. If you are not actively doing I/O operations against the SR, you won't see anything. XenCenter (at least the free one) doesn't seem to make sure that the iSCSI SRs it thinks are connected are still connected.

So for a simple example, my example pool has 2 hosts in it. I have one Windows vm using the iSCSI shared storage. If I turn the Windows vm off, and then shut down the Openfiler host, XenCenter doesn't notice.

"Look Ma, no Openfiler, no warnings!"

Of course, in real usage, that wouldn't happen very often. You are usually using the SR, that's the whole point of having one.

You'll first notice that something is wrong if you attempt to start the vm on the iSCSI SR (which of course triggers some I/O attempts). These errors are posted to the log for the vm, not the SR. The error is not particularly clear in what's going on:

The SR backend failed to complete the operation
Errors in the vm log

Something like "SR unavailable for writes" or "SR unreachable" would be nice. A big red splat (red X) on the SR would be nice too. Even after you trigger some SR errors, the errors don't appear in the SR log.

Nothing in the SR log

So be aware that in some cases, XenCenter is blissfully unaware of the status of its SRs.

Personal tools