### Human: Can you tell about Migration of resource heavy VMs might get stuck on AHV clusters. ### Assistant: On AHV clusters, migration of large VMs might fail. In the Prism web console, you can see the migration progress up to 99%, and then the migration stops indefinitely.. VM live migration can be manually initiated or be a result of putting the host into maintenance mode (to perform firmware or hypervisor upgrade, for example). Live migration of the VM that has a high rate of change to its memory contents may take a significant amount of time or look "stuck" and the process eventually fails. This can happen on a cluster where 1G and 10G uplinks are configured. In both cases, this happens when the change rate of the VM memory exceeds the network bandwidth.

Prism will generate an alert when the migration task fails. Furthermore, sample errors can be found in the acropolis.out log on the Acropolis leader. Follow the steps from KB 2305 to find the Controller VM (CVM) with the Acropolis leader role.

The following error messages are commonly seen in the acropolis.out log on the Acropolis leader on clusters configured with 1G uplinks:

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error

The following error messages are commonly seen in the acropolis.out log on the Acropolis leader on clusters configured with 10G uplinks:

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error. Perform the following steps to investigate the issue:

Check maximum available NIC bandwidth and load balancing mode:
	Run the following commands on CVM:
		nutanix@cvm:~$ manage_ovs show_uplinks
		In the output of commands mentioned above, check which NICs are assigned to br0 bridge and their speed:
		nutanix@cvm:~$ manage_ovs show_uplinks
		
	Check VM migration progress by running the following command multiple times with a delay of 1-2 minutes on the source AHV host:
	nutanix@cvm:~$ manage_ovs show_uplinks
	To identify the network bandwidth utilization, you can check the sar stats for the AHV source host.
	For example, where VM migration failed on 1st Feb at 08:43 AM, the following command can be run on the sar file. The command is filtering occurrence where physical NIC is transferring more 8Gb per second of data.

	nutanix@cvm:~$ manage_ovs show_uplinks

	Starting from AHV 20201105.2159, in /var/log/libvirt/qemu/<vm_uuid>.log on the source host, you can also see the memory dirty rate of the migrated VM which is helpful when troubleshooting failures:

	nutanix@cvm:~$ manage_ovs show_uplinks
	

For further troubleshooting, consider checking /home/data/logs/acropolis.out on the Acropolis leader. KB 7949 describes the normal VM Live Migration Process and can be used as a reference.

Workarounds to consider:

Power off memory-intensive VMs and use acli to start it on the target AHV host instead of live migrating the VM. Run the following command on any CVM in the cluster to start VM on a specific AHV host:
	nutanix@cvm:~$ manage_ovs show_uplinks
	Schedule the VM migration or maintenance at a time when the VM is as idle as possible.Use faster uplinks if available.Verify that the network uplinks are not degraded.

Note: Use of LACP load balancing mode may not improve the situation as a single TCP connection can only use the bandwidth of a single NIC.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.