User Login is very slow on Virtual Machines connected to Broadcom Network cards !

A weird behavior was noticed on few Hyper-V host machines, these machines are mainly Dell Rack servers as the powerful R720 Model. These servers came with mix of Network adapters (based on your configuration) as the Broadcom built in adapters and extra Intel cards. The behavior noticed was that Virtual machines that are connected to the Broadcom NICs has poor performance and user login takes several minutes on these specific machines. VMs connected to the Intel Network card login normally and has better performance.

When I moved the VMs connected to the Broadcom NICs to the Intel NICs they behaved normally and users can login much faster. I then tried to move the VMs on the Intel Cards to Broadcom and we got the poor performance. We concluded it must be something related to the Broadcom Network card.

Troubleshooting steps:

1. Updating the Broadcom NICs to the latest driver. Still No change.
2. We compared the options/Properties of Intel Card vs. Broadcom card and it was noticed that Transmit buffers in Broadcom was 200 vs 500 in the Intel cards, I tried changing it to match the Intel settings. Still No change.

Resolution:

After several investigations, the problem turned out to be in the VMQ setting which was enabled on both Broadcom and Intel. When it was disabled on the Broadcom card, the VM worked perfectly. Looks like the Broadcom card is not compatible with the VMQ feature.

Reference Links:

1. High Ping Latency in Hyper-V Virtual Machines: http://www.flexecom.com/high-ping-latency-in-hyper-v-virtual-machines/
2. Slow Network Access Within Virtual Machines - Broadcom and Hyper-V: http://fundamentallygeek.blogspot.com/2012/11/slow-network-access-within-virtual.html
3. VMQ feature http://technet.microsoft.com/en-us/library/gg162704(v=ws.10).aspx

Read More

How to Update Lync 2013 Standard Backend Database ?

Microsoft Releases frequent Cumulative updates for Lync 2013 servers and Most of these updates requires updating the Lync 2013 SQL Backend database. In this article I will discuss how to update the SQL Express backend database after installing a new Lync 2013 Cumulative update or specific Rollup that requires DB version update.


First to check the status of the Database after installing the cumulative update and whether you need an update, you need to run the below two commands from elevated Lync Power Shell.

1. PS C:\> Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn servername.domain.com | FT DatabaseName, ExpectedVersion, Installedversion        
2. PS C:\> Test-CsDatabase -CentralManagementDatabase | FT DatabaseName, ExpectedVersion, InstalledVersion

The Output will be something similar to that (This is the output of the second command on one of my deployments)

DatabaseName        ExpectedVersion      InstalledVersion

     ------------                   ---------------              ----------------

         xds                             10.13.2                    10.13.1
          lis                                 3.1.1                      3.1.1

 

As you can see there is a difference between the expected and installed value in the XDS database.

 

To update the Database we need to run the below commands in sequence:

 

1. Install-CsDatabase -ConfiguredDatabases -SqlServerFqdn FEBE.FQDN -Verbose
2. Install-CsDatabase -CentralManagementDatabase -SqlServerFqdn CMS.FQDN -SqlInstanceName DBInstanceName -Verbose

 

 

After that we need to enable CsMobility service and Run Bootstrapper as follows:

 

1. Enable-CsTopology
2. Bootstrapper.exe
3. Reboot the server.

 

This should take care of updating the backend DB and you can double check by running the previous Power Shell again and ensure both expected and installed versions are the same.

Reference Links:

Lync 2013 Cumulative Update List

http://blogs.technet.com/b/nexthop/archive/2013/03/26/lync-2013-cumulative-updates-list.aspx

Lync Server Resources

http://blogs.technet.com/b/nexthop/p/links.aspx


 

Read More

UAG 2010 SP4 Released to support Windows 8.1 and IE 11

Good news for all Microsoft UAG 2010 users, Service Pack 4 (SP4) was released officially yesterday to support Microsoft Latest Operating system Windows 8.1 and Internet Explorer 11 (IE 11). This new SP4 provides other important features as the support of Remote Desktop Connection RDC 8.1 and the Remote Apps published from Remote Desktop session Host on Windows 2012 or 2012 R2.

The Link to download the Latest SP4 is as follows:

https://www.microsoft.com/en-us/download/details.aspx?id=41181


For More details about features and fixes included in this Service Pack, please check the following link:

https://support.microsoft.com/kb/2861386


Service Pack 4 includes some other fixes as well as stability and performance enhancements, I will give it a try and hopefully it will be smooth operation.

Read More

Why DPM can only take offline backup for some VMs under Hyper-V 2012R2 Host ? Need SCSI !!

I published a recent article on migrating/moving VMs from 2008R2 to 2012R2 Hyper-V host (you can check it at http://itcalls.blogspot.com/2013/11/how-to-migratemove-virtual-machines.html ), after successfully moving these VMs we faced another problem trying to backup them. We were using the latest DPM 2012R2 and we noticed that it added these VMs as offline only and for some reason it can't take online backup of them.

After some investigation we noticed an error on the hyper-v host with Event ID 10103 (check below image) which clearly mention that backup will fail because this VM doesn't have a SCSI controller.

So the solution was just to add a SCSI controller even if its connected to nothing, after that online/hot backup was taken smoothly without any problem.

So what was the problem ?

I discussed this issue with several Microsoft support personnel and It turned out that online or hot backup for a VM in 2012R2 Hyper-V host requires mounting a new VHD in the VM and then dismounting it later. Since only the SCSI controller can mount/support hot plugging of virtual disks, it became clear why we need this SCSI controller.

Old version of Hyper-V didn't work this way,  it required that the Hyper-V host mount the guest VHD as part of the backup process which is something Microsoft didn't like as it increase the surface of attack on this host.

Read More

How to Migrate/Move Virtual machines from 2008R2 Host to 2012R2 Host ?

We were recently working on a project with my team to move and migrate several 2008R2 virtual machines from 2008R2 SP1 Hyper-V Host to the latest Hyper-V 2012R2 host and we faced a lot of troubles since the old import and export is not working anymore and not supported.

Hyper-V Virtual machines that used to be exported from Server 2008R2 Host were utilizing the version 1 WMI namespace which resulted in an export file (.exp) which was used to represent the exported virtual machine. When Server 2012 was released version 2 of WMI namespace was introduced and version 1 was deprecated (Server 2012 still compatible with old version 1 WMI namespace but no new features or additions will be added, Microsoft normally use deprecated as a step to fully remove and delete this feature on the next version) and then it was totally deleted in Server 2012R2.

For a list of deprecated and removed features in 2012/2012R2, check this link

http://technet.microsoft.com/en-us/library/dn303411.aspx

After several trials and investigations done by my team we reached two recommended methods for moving the VMs to Hyper-V 2012R2 hosts (Both were confirmed later by Microsoft Support Team) as follows:

1. The Easiest way will be to turn off the VM on the hyper-V 2008R2 host and stop the VMMS service on the host server as well to unlock all VM files. The next step is to copy all Virtual Machine files/folder including the VHDs, XML..........etc. to the server 2012R2 and import them directly.
2. Another method can be done is to use an intermediate server (this can be used if you already exported your VM and deleted the original VMs). In this method you will export the VM from the 2008R2 host and then import in in 2012 Hyper-V host (no need to start it up) and then export it again from the 2012 Host and import it in 2012 R2 host.

For more information about Version 2 WMI Namespace, check the below link

http://blogs.msdn.com/b/virtual_pc_guy/archive/2012/05/30/the-v2-wmi-namespace-in-hyper-v-on-windows-8.aspx

An excellent resource on Hyper-V generation 2 virtual machines is the 10 blog series by John Howard

http://blogs.technet.com/b/jhoward/archive/2013/10/24/hyper-v-generation-2-virtual-machines-part-1.aspx

Read More

WMI Unhealthy on 2008R2 Domain Controllers - WBEM_E_QUOTA_VIOLATION

Windows Management Instrumentation (WMI) is a key core windows management technology. It provides a consistent approach to carry day to day management operations with programming or scripting languages.

I recently started getting WMI failures on daily basis on my 2008R2 domain controllers accompanied by several scripts failure and DNS performance degradation.

Also I noticed that the Configuration Manager SCCM evaluation rules on this domain controller failed and SCCM is reporting errors. The policy Request date on the SCCM is few hours back and it will never report back to SCCM till the DC/Server is rebooted.

Troubleshooting Steps:

1. I started by running the WMI diagnosis tool from http://www.microsoft.com/en-us/download/details.aspx?id=7684
2. The WMI diag log file reported WBEM_E_QUOTA_VIOLATION as follows:

.5265 16:34:02 (0) ** 981 error(s) 0x8004106C - (WBEM_E_QUOTA_VIOLATION) WMI is taking up too much memory

.5266 16:34:02 (0) ** => This error is typically due to the following major reasons:

.5267 16:34:02 (0) **    - The requested WMI operation is extremely costly in terms of resources and

.5268 16:34:02 (0) **      the WMI provider handling this operation has exceeded the authorized limits.

 3.  tried later to check whether the basic WMI function is working by running the below test:

1.     From Elevated Command Prompt type Run wbemtest, connect the namespace root\cimv2

2.     Click Query… and enter the following query “Select * from Win32_ComputerSystem”

3.     This test failed and the following error was reported.

0x80041017 Facility: WMI  Description: Invalid Query

1   4. I tried fixing and rebuilding the WMI Repository as follows:

• Disable and stop the WMI service. sc config winmgmt start= disabled and net stop winmgmt
• At a command prompt (cmd), change to the WBEM folder. cd %windir%\system32\wbem
• Rename the repository folder. rename repository repository.old
• Re-enable the WMI service. sc config winmgmt start= auto
• Run the following command to manually recompile all of the default WMI .mof files and .mfl files
• cd %windir%\system32\wbem
• for /f %s in ('dir /b *.mof *.mfl') do mofcomp %s

The only way to get around this issue was to manually reboot the server. After Rebooting the server, it works for  few hours without a problem then the failures start again. One thing else to be noticed is that the WMIPRVSE.exe process is consuming huge amount of memory during this problem.

Resolution Steps:

1. Increased the "MemoryPerHost” value to 1 GB (1073741824), by default it should be 536870912 which means 512 MB as per attached article

Memory and Handle Quotas in the WMI Provider Service

http://blogs.technet.com/b/askperf/archive/2008/09/16/memory-and-handle-quotas-in-the-wmi-provider-service.aspx

2. Install the following WMI fixes

KB Article Number (s) : 2705357  

Language: All (Global)  

Platform: x64  

Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix399300/7600/free/447586_intl_x64_zip.exe)

KB Article Number (s) : 2692929  

Language: All (Global)  

Platform: x64  

Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix395847/7600/free/446374_intl_x64_zip.exe)

KB Article Number (s) : 2617858  

Language: All (Global)  

Platform: x64  

Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix384504/7600/free/437954_intl_x64_zip.exe)

KB Article Number (s) : 2465990  

Language: All (Global)  

Platform: x64  

Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix354372/7600/free/425609_intl_x64_zip.exe)

KB Article Number (s) : 2492536  

Language: All (Global)  

Platform: x64  

Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix360823/7600/free/429002_intl_x64_zip.exe)

For a list of suggested WMI hotfixes on different windows platform, please check this blog which is maintained and updated regularly.

http://blogs.technet.com/b/askperf/archive/2011/08/05/suggested-hotfixes-for-wmi-related-issue-on-windows-platforms.aspx

Read More

AD CS not configured for Revocation checking of all certificates

Recently the SCOM server (One of your best friends on the network) started reporting the error "AD CS not configured for Revocation checking of all certificates", SCOM reported Event 128 as a warning

Event 128 is normally reported when someone try to use Certificate Request with all time-valid CA certificates to request a certificate. However y default CA doesn't support such request and event 128 gets reported.

After checking this issue and consulting Microsoft tech support, this issue will normally occur only after renewal of CA certificate. When the CA certificate is renewed, the OCSP Response signing certificate used for validation of existing certificates must still be signed by the CA certificate that was used to issue the existing certificates and new CA certificate. However by default CA doesn't support the renewal of OCSP Response signing certificate by using a previous CA certificate.

This issue/behavior can be fixed as follows:

1. From elevated CMD run the certutil -setreg ca\UseDefinedCACertInRequest 1
2. Restart the CA services

This command will enable the CA support for certificate request signed by old certificate. If the OCSP is not renewed, you need to go ahead and renew it as per the following articles

http://blogs.technet.com/b/xdot509/archive/2013/06/06/operating-a-pki-ca-certificate-renewals-and-ocsp.aspx

http://technet.microsoft.com/en-us/library/cc770413(WS.10).aspx

Read More