Search This Blog

Sunday, May 29, 2011

Investigating network devices and device settings as a cause of VMware Converter failures

Investigating network devices and device settings as a cause of VMware Converter failures

Symptoms

  • Conversion process is extremely slow
  • Very slow network throughput
  • Conversion fails before completing
  • Conversion fails between 2% and 96% during the cloning process
  • You may receive the following errors:
    • Image processing task has failed with MethodFault::Exception: sysimage.fault.DiskLibConnectionFault
    • Image processing task has failed with MethodFault::Exception: sysimage.fault.NfcConnectionFault
    • Image processing task has failed with PlatformError fault: 5666
    • Image processing task has failed with PlatformError fault: 79394
    • Image processing task has failed with PlatformError fault: 4898
    • [NFC ERROR] NfcNetTcpRead: timed out waiting for data
      [NFC ERROR] NfcFssrvrRecv: failed with code = 9
      [NFC ERROR] NfcFssrvr_DiskIo: failed to receive io reply
      DISKLIB-LIB : RWv failed ioId: #1 (5666) (34)
      NfcNetTcpWrite: timed out waiting for socket to be ready (101)
    • Error 5666 writing to the destination volume
    • [NFC ERROR] NfcNetTcpWrite: bWritten: -1 err: 10053
      [NFC ERROR] NfcFssrvrSend: failed
      [NFC ERROR] NfcFssrvr_Close: failed to send close message
      [NFC ERROR] NfcNetTcpWrite: bWritten: -1 err: 10053
    • Error 79394 writing to the destination volume
      DISKLIB-LIB : RWv failed ioId: #50301 (79394) (34)
      [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
    • NfcNetTcpRead: timed out waiting for data
      NfcNetTcpWrite: timed out waiting for socket to be ready (101)
      [NFC ERROR] NfcFileSendMessage: hdr send failed:
      [NFC ERROR] NfcFile_Send: Something is messed up
      [blockLevelCloningTaskImpl,831] File transfer failed.
      Dst error: Network error -- Failed to send header message:
    • Error E7D1001E: Unable to read from file. Error 00000040: The specified network name is no longer available.
      Failed to write out header: The specified network name is no longer available (4194313).
      File transfer failed.
      File error -- Failed to delete file: File not open

Purpose

To verify if the configuration of network interface cards (NICs), switches and switch ports, firewalls, and load-balancer devices are causing problems when using VMware Converter's cloning process.

Resolution

During the cloning phase of conversion, data is being transferred from the source computer or virtual machine to the destination.  A break in the transfer of data caused by network errors or improper network configuration will cause failures or very slow conversion performance.

Determine if network device settings are interfering with the conversion or cloning process

To determine if network device settings are interfering:
  1. Verify that there are no host network interface cards (NICs) or network devices in the VMware Converter environment that have been statically configured to be at a different speed or duplex. This includes settings on the source operating system, switches and networking devices between the source and destination and the destination server.
    If the destination in an ESX server, verify the speed and duplex setting using the Virtual Infrastructure client: click Configuration > Network.  Observe the settings for the VMKernel virtual switch.  For more information see: Configuring the speed and duplex of an ESX/ESXi Server host network adapter (1004089).
  2. Make sure you check the physical speed indicator lights on the NICs and switches and verify that each end has a corresponding speed and duplex.  In general ensure all devices are using auto negotiation. Check this setting in the operating system or drivers, and then verify the operation on the indicator light on the device. If you must use manual or static settings, ensure that the setting is not higher than both the device itself and the device it is connected to (peer) is capable of operating.Note: In some cases, auto-negotiation may fail. In this case use static configuration on both sides of that network segment. Be sure you set corresponding duplex and speed settings between the source and its peer to avoid a duplex mismatch condition. See: Duplex Mismatch (http://en.wikipedia.org/wiki/Duplex_mismatch).

Determine if the failure is being caused is a problematic NIC, network device, or cable

The easiest approach is by simplifying the networking environment (elimination). Gradually reducing the number of network devices and network segments involved in the conversion, verify speed indicators, and swap for known good equipment and cables.
 
To determine the cause of the failure is a NIC, network device, or cable:
  1. If applicable, convert directly to the ESX Server rather than via VirtualCenter. If converting to a standalone virtual machine file over a network share, try performing the conversion to a locally attached drive that is not included in the source of the conversion.
  2. Bypass or disable any network load-balancers between the two end-points.
  3. If you are using a security firewall or stateful packet inspecting (SPI) firewall, make sure the connection is not being blocked as malicious traffic. You may need to review the alerts or logs to determine this.
  4. Try connecting the source and / or destination servers to a different port on the switch. Some switches may have failing or faulty ports while other ports work properly.
  5. If you use a managed switch, check the administration console for logged connection errors or corrupt packets.
  6. Use a dedicated switch rather than a shared network switch between the source and destination.
  7. Swap the network cables between the switches or servers with known good cables.
  8. Use a crossover cable between the source and destination. When using this configuration make sure the speed and duplex indicators on the physical hardware correspond between the two systems.
  9. Bypass network conversion. Create a standalone virtual machine on the source operating system's file system, and the copy or convert it manually to the destination.
    Note: If you are using ESX server as the destination, you must use vmkfstools -i to import the virtual machine.

No comments:

Post a Comment