Hyper-V 2012R2 failing network connectivity using fully converged networking SOLVED!


Update: The real fix is not to disable VMQ on the Management vNIC but rather fix the MAC address issue itself. I did that as well, you can find it here!

At least in my case that is 🙂

On a project I’m currently working on I am using SCVMM 2012R2 to bare-metal deploy Hyper-V 2012R2 on HP DL380P gen8 / BL460C gen 8 system.

The DL380P are equipped with 530FLR 2port 10GBe (Broadcom BCM57810) and the BL460C are equipped with 554FLB and 554M (both Emulex) adapters. I won’t go into too much details about the setup but basically it comes down to deploying a logical switch on a team and attaching the management, live migration and cluster network adapters to it.

While testing this setup out I was confronted by erratic network connectivity issues. Cluster validation wasn’t able to connect via WMI (so effectively, no cluster could be build), RDP would drop sporadically, etc. I found some excellent blogs about these issues on hyper-v.nu I thought my issues where related to. For more details see: http://www.hyper-v.nu/archives/hvredevoort/2013/12/november-2013-ur-kb2887595-causing-network-problems/.

In my case I also went through a process of elimination as described in these blogs. Eventually I wound up disabling VMQ on the teamed network adapters and everything worked fine, except of course this isn’t a viable solution (you might as well rip those 500 dollar NICs out and replace them with Realtec’s right 😉 ).

So I wasn’t happy about it and theorized for a bit after reading this excellent series of posts http://blogs.technet.com/b/networking/archive/2013/09/10/vmq-deep-dive-1-of-3.aspx (2 and 3 can be found here as well). Long story short, using VMQ will create queues based on the MAC address (imagine it to be a sort of a routing table). When you deploy a Hyper-V host without dedicated management adapters you will end up with 3 (v)NICs that have the same MAC address, the multiplexer NIC will inherit the MAC from the physical NIC, the virtual NIC which is created afterwards will inherit the MAC from the multiplexer NIC. By default the port profile for host management will have VMQ enabled.

Looking at the queues post deployment you will see something like this:

vmqbefore

As you can see, there are 2 queues mapped to the same MAC address.

After disabling VMQ on the port profile:

portprofile

And remediating the host you will end up with:

remediate

Voila! No more “routing” issues.

vmqafter

Since there is no need for VMQ on the management adapter, this is a viable solutions.

Hope this helps!

Advertisements

4 thoughts on “Hyper-V 2012R2 failing network connectivity using fully converged networking SOLVED!

  1. […] This blogpost by Ben Gelens describes the same issue. Ben solved it by disabling the  Virtual Machine Queue (VMQ) on just the management nic. […]

  2. […] 2012R2 failing network connectivity using fully converged networking SOLVED! | MS Sec by Ben: https://mssecbyben.wordpress.com/2014/01/03/hyper-v-2012r2-failing-network-connectivity-using-fully-c… And review this KB Poor network performance on virtual machines on a Windows Server 2012 […]

  3. Fernando says:

    Hello and thank you for this information!! we are suffering a lot of problems. We have disabled the option “Enable Virtual Machine Queue” in SCVMM as you told us. But, we cannot mark in “Remediate” option. This option is disabled to push. Is the fix applied without mark in Remediate?
    thank you in advance.

    • bgelens says:

      Hi Fernando,
      This could happen if no VMQ Queue was assigned to the vNIC in the first place. You should check if VMQ is enabled and used at all on the Hyper-V host. You could do this via PowerShell Get-NetadapterVMQ and Get-NetadapterVMQQueue. VMQ is only enabled by default if the NICs bound to the VSwitch are reported as 10Gb and the NIC supports it. If you receive output from previous command you should check the VMQ weight on the vNIC. You can do this using PowerShell as well. Get-VMNetworkAdapter -ManagementOS -Name MGMT | select vmq*. If the VMQWeight is set to 0, it means your port profile setting set on the SCVMM side is already effectuated. If it’s set to 100, it probably means SCVMM is not aware of the latest status of the Hyper-V host and you should trigger a manual refresh for the remediation button to become active. Hope this helps you. For more info about the VMQ issue in general and how it is actually solved without the need for workarounds please read this blog by my colleague Hans Vredevoort here: http://www.hyper-v.nu/archives/hvredevoort/2014/12/another-update-on-the-vmq-issue-with-emulex-and-hp/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: