HP Blades, Virtual Connect & ESX considerations

I may have already mentioned that one of our projects for the year is to transition our corporate ESX cluster from 2U hardware onto blades.  The process of transitioning does not come without some concern and some caveots moving to the blade architecture.  We feel that blades are a good fit in our case for this particular cluster (we run several ESX clusters).   Our VMware View deployment is our first production ESX workload on blade hardware.  We have learned a few things from this deployment that might be helpful.

Multiple VLANs on a single NIC
When using Virtual Connect, the default configuration sets VLAN tagging support to “Tunnel VLAN Tags.”  The mode is self-explanitory and just means that the “Ethernet Network” chosen and assigned to the server profile in Virtual Connect is the only network visible to the blade.  For most blade users, this setting works fine and many ESX deployments might be ok with this configuration.  But for many ESX deployments, people require multiple VLANs to be brought up on the same physical NIC.  The “Tunnel VLAN Tags” mode does not allow for this functionality.  

To allow for multiple VLANs on a single NIC, you must login to Virtual Connect, expand Ethernet Settings, select Advanced Settings and change the VLAN Tagging Support from Tunnel to “Map VLAN Tags.”  Map VLAN Tags exposes a Multiple VLANs option in the network assignment drop-down under your server profile.  Once you select Mutliple VLANs, a new window appears and you may select as many VLANs as you need exposed to the server.  The ESX host is then required to tag its traffic on these NICs.

Number of nodes per enclosure
Thou shalt not run more than 4 nodes of an ESX cluster in the same blade enclosure.   No, its not the 11th commandement, but it is an important rule to know.  Thanks to Duncan Epping’s article on the topic, we discovered a major implementation hazard for ESX on blade architecture that we didn’t have to experience first hand.  We did have our own enclosure failure, which made us aware that we could have been affected, however.  The pitfall is that HA has primary and secondary nodes in a cluster.  An ESX HA cluster can have up to five primary nodes, but never more.  The first five nodes in the HA cluster become primaries and these roles never get reassigned if a primary node fails.  The primary nodes are responsible for directing the HA activites.  So, you don’t want all your HA primary nodes running in the same enclosure.  If all five HA nodes are running in the same enclosure and it fails, you will not get the desired HA restarts on the other ESX nodes in your cluster.  Duncan’s article gives a great overview of the HA clustering architecture and sheds light on a little known consideration.  

Service Console & VMotion
Perhaps the favorite thing about Virtual Connect and ESX is the ability to creatively configure Service Console and VMotion using just two NICs, and providing redundancy and 
isolation as needed for these functions.  Lets look at the best practices:  

  1. Service Console should have two NICs teamed for redundancy of this network link.
  2. VMotion should have its own dedicated NIC for the best performance of VMotion traffic.

So, what makes Virtual Connect suit this well?  
First, Virtual Connect is redundant on the VC-Ethernet side.  We can create a single “Shared Uplink Set” with both Service Console and VMotion tagged traffic.  The stacking link between the two VC-Ethernet module will provide for traffic on NIC0 to be rerouted on Bay 2 if the uplink on Bay 1 is down.  As long as both VC-Ethernet modules are functioning, the stacking links would be utilized.  

Second, we can use ESX NIC Teaming failover settings to keep the traffic separate, except when a failure occurs.   If you lost a VC-Ethernet module, your Service Console and VMotion traffic would be failed over using ESX onto the same NIC on the other VC-Ethernet module.  

There are really a lot of options in this space and this is my high-level implementation.  I think its a great solution and can’t find many trade-offs for this.  In an blade environment where NICs are a premium, this is a wonderful solution. 

(By the way, I didn’t devise this SC/VMotion configuration – its something someone else posted, but I can’t find the original blog post to give you credit… If it was you, please let me know.)

Tags: , , , ,

 

About the Post

7 Responses to “HP Blades, Virtual Connect & ESX considerations”

  1. M #

    Hya,

    We currently have the “Tunnel VLAN Tags” option enabled, and we have created networks that comprise of multiple Vlans. Still our ESX’es can talk to eachother of each Vlan, do the ESX’es use a different NiC per Vlan ? (The vSwitches have both nic’s in them)

    greetz!

    M

    March 10, 2009 at 12:17 pm Reply
    • Philip #

      M – That would be my guess. We tried to get multiple VLANs across a single NIC with Tunnel mode, but it would not work on our enclosures. We had to enable the Map VLANs mode to get multiple VLANs across a single NIC. Bringing a single VLAN per NIC to the ESX host will certainly work. We have other clusters where we are doing that with traditional rack-mount servers. In a simple network where all your VM’s are on the same VLAN, it may make sense to do this on blades, too. Our deployment is more complex with many VLANs for our VM’s. In a blade configuration with 6 total NICs possible, we would be limiting our capabilities if we did not do the tagging. Hope this helps…

      March 10, 2009 at 1:59 pm Reply
  2. Bobby McIntire #

    Philip,
    Just to clarify. We have 6 virtual connect ethernet modules set up with 1 set of fiber connected to a Cisco switch as port channels. Each port channels are set to pass all VLANs. On the ESX side I have 6 nics in a BL460 server. 2 in one Virtual Switch 1 for Service console and vmotion and 3 in Virtual Switch 2 for all Virtual machines and 1 in Virtual Switch 3 for ISCSI (iso and templates). Virtual connect is set up to use Tunnel VLAN Tags. Will we be only able to pass 3 vlans since there are 3 nics on the Virtual Switch 2 running all the virtual machines? We have over 15 VLANS that need to be passed for thru the Virtual Switch 2.

    Great blog,

    Bobby McIntire

    April 21, 2009 at 10:44 am Reply
    • Philip #

      Bobby,
      No, I think you’ll be able to pass as many VLANs as you configure onto the 3 physical ports connected to vSwitch2, but you’ll need to be in map vLAN mode. When you change your Virtual Connect option, it exposes a “Multiple VLANs” selection which lets you choose which VLANs you want to carry across those ports. You have the capability, you just need a small configuration change. This configuration change does not affect any of your other server profiles that are already created. It just exposes the “Multiple VLANs” capability in a server profile.

      In tunnel VLAN mode, you are only capable of carrying a single VLAN at a time (as far as I can find anywhere). I wouldn’t recommend tagging 3 different VLANs, one on each physical port and then using all 3 in the same vSwitch. I don’t think that’s a good practice in VMware. I’m not sure that scenario would work at all.

      April 21, 2009 at 1:51 pm Reply
  3. Ray #

    Enabling Tunnel VLAN Tags (even according to the Virtual Connect help file) will pass all VLAN tags through the VC domain unmodified. It’s ESX that will then manage the various VLAN traffic dependent on your virtual switch configuration. If you have VLAN tunneling enabled and attempt to configure shared uplinks, only untagged frames will be transmitted (tagged frames will be dropped).
    If you enable Map VLANs option, any dedicated links (ie individual ports that are not members of shared uplink sets) can transmit untagged frames only – all tagged VLAN traffic on dedicated links will be dropped. And you’ll have to map the traffic from the shared uplink sets to the appropriate NICs/server profiles. This is a lot of work – ESX will do it for you if you just pass the traffic with VLAN Tunneling (tagging – 802.1q).
    Check out the HP Virtual Connect Cookbook:
    http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c01471917/c01471917.pdf
    Scenarios 7 & 8 specifically detail ESX config.

    April 23, 2009 at 12:08 pm Reply
    • Philip #

      Ray – Thanks! I think I have figured out where we ran into problems in our configuration. We had a shared uplink set configured and were attempting to use it instead of simply defining an Ethernet network. I re-read the documentation (btw, HP released a brand new cookbook this week – tr.im/jwf5), and I now see that we should omit the shared uplink for ESX to work correctly in Tunnel mode.

      April 23, 2009 at 1:43 pm Reply
  4. Bobby McIntire #

    Philip/Ray,
    I guess I’m a little confused. I had my enclosure set up to use Tunnel VLAN tags and we were unable to get a VLAN 100 to pass and communicate correctly. I thought it had to do with the single vlan per nic as described above. According to the network guys, the Tunnel Uplinks that are set up to each Virtual connect in the back of the Enclosure should pass all VLAN. I guess I’m not sure how it should be set up. The current enclosure was working well but we only have a few vlans. I brought up a new server on the 100 vlan and it couldn’t communicate. Any help would be appreciated.

    April 28, 2009 at 8:54 pm Reply

Leave a Reply

%d bloggers like this: