### Human: Can you tell about Generic AHV troubleshooting scenarios. ### Assistant: This article provides an overview of generic AHV troubleshooting scenarios.. This article provides a generic description of common issues that can be seen on clusters running AHV and propose troubleshooting methods.

Common scenarios:

Unreachable hostHost in a crash loopUnexpected host rebootUnexpected guest VM rebootInability to perform VM management actionsStuck VM migrationFailed VM live migration due to Jumbo frames misconfiguration.. Logs are the most important source of information when performing troubleshooting. You can find below the list of logs on the CVM (Controller VM) and on the AHV host, which should be checked during troubleshooting.

CVM logsThe most important log file on a CVM to check while troubleshooting AHV related issues is /home/data/logs/acropolis.out as the Acropolis service coordinates most of the activities. Always check acropolis.out on the CVM hosting the Acropolis leader. KB 2305 describes steps on how to find the Acropolis leader.

To find which CVM is the Acropolis leader from logs or to show leader changes if any have occurred recently, run the following command:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Example:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

If the log file on the Acropolis leader does not contain relevant details, consider checking acropolis.out on other CVMs.

Note: acropolis.out is updated with new events together with any action against user VM. Lack of updates means the service may not be healthy on the CVM.

Host logs

Important logs are:

Log fileDescription/var/log/libvirt/libvirtd.logEvents related to libvirtd service/var/log/libvirt/qemu/NTNX-<HOST>-CVM.logThe QEMU log for the CVM on this host./var/log/libvirt/qemu/<vm uuid>.logEvents related to VM lifetime (start, stop, migrate). To find a specific VM's UUID, use acli vm.list from a CVM and take note of that VM's UUID. You can find out which host that VM is currently on by checking the output of acli vm.get <VM_NAME> and taking note of the host_name field./var/log/libvirt/qemu/hook.logContains some basic information on calls that Acropolis makes into QEMU/var/log/acropolis_ovs.logHost-level OVS-related changes and queries from the Acropolis service are logged here on each host. You should see logs when new taps are created, changed, or destroyed for VMs running on this host.ovs-ctl.log, ovsdb-server.log, ovs-vswitchd.log in /var/log/openvswitchEvents generated by ovs-vswitchd service/var/log/iscsi_redirectoriSCSI Redirector process is what manages iSCSI connectivity between the user VM's vdisks on AHV hosts and the Stargate processes on CVMs. iSCSI redirector is what AHV uses to redirect vdisks to be hosted by other CVMs when the local CVM is inaccessible via iSCSI (e.g. CVM is down or Stargate is having issues)./var/log/NTNX.serial.out.0The CVM's console log. If a CVM is powered on, but not reachable via SSH or is in a boot loop, you can check this file to view the CVM's console output.

Note: Always check the timezone configured on CVM and AHV hosts while correlating events. The default timezone on the AHV host is UTC and the default timezone on CVM is PST. The timezone on the AHV host should not be modified. Refer to the Nonconfigurable AHV components section of the AHV Administration Guide. If the timezone on the AHV host was changed, it will be automatically reverted back to UTC on the next AHV upgrade.

Scenarios

1. Unreachable host

A host can become non-accessible due to networking issues or due to the inability to boot the hypervisor.

1A. Host unreachable due to networking issues

Perform the following tests to identify the scope of the issue:

Does the host respond to pings from another host in the same subnet (other AHV hosts for example)?Does the host respond to pings from another host in a different subnet?

If an AHV host responds when queried from hosts in the same subnet but does not respond to hosts in other subnets, make sure that routing is correctly configured on network devices.

If connectivity to the host is intermittent, make sure that the host's IP address is not used by other devices on the network. Perform steps described in the Solution section of KB 3025 to host with the conflicting IP address.

If a host is not reachable, check if the IPMI web interface is available. If not, physical access to the server is required to check power status and network cabling.

If you can log in to the AHV host via IPMI console, perform basic network troubleshooting:

Check the configuration of the br0 bridge and make sure it is correct:
	nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
	Check br0 configuration:
	From AHV host, connect to CVM:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		From CVM, run the following commands:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		In the output of commands mentioned above, make sure that correct NICs are assigned to br0 and other bonds. For example:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		Update bridge configuration by following steps from KB 8835.Check which NIC is active if active/pass mode is used by following the steps described in How to select active NIC in active/passive bond section of KB 2090.
	

1B. Host unreachable due to inability to boot the hypervisor

If the connection to the IPMI interface is working fine, then open the server console and check the status of the host. Following scenarios are possible:

The host is powered off. Power it on remotely via IPMI.The host is powered on but stuck during BIOS initialization. Check the hardware event log for hardware-related issues.The host is powered on but is in a crash loop (scenario #2 below).

2. Host in a crash loop

If the host is constantly rebooting, try making the screenshot of the call stack shown on the screen and search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support at https://portal.nutanix.com/.

3. Unexpected host reboot

If the host was unexpectedly rebooted, perform the following checks to scope the issue:

Were there any upgrades performed on the cluster at the same time?Were any events happening in the environment, like power or overheating issues, that could affect the hardware? Check the IPMI event log to verify it.

Collect basic information about host uptime, last reboot date:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Check IPMI event log for hardware errors. In addition to this, check contents of /var/log/mcelog file if it exists. Make sure that the timestamp recorded there matches the date and time of the unexpected reboot.

Sample:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Run ahv_crash_file_check NCC check to verify if any core dump files are present:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

More details about this check can be found in KB 4866.

If ahv_crash_file_check finds any issues, /var/crash folder on the affected host will contain one or more subfolders with the date and time of crash specified in the name.

Each of these subfolders may contain one of the below:

vmcore and vmcore-dmesg.txt if core dump was successfully created.vmcore-dmesg-incomplete.txt if core dump creation was aborted due to some external factor (like lack of free space for example).

Review vmcore-dmesg.txt and vmcore-dmesg-incomplete.txt files.

Check /var/log/messages and messages.<number>.gz files for any critical events preceding host reboot.Check if the AHV host abruptly rebooted. It generates a series of "recovering journal" and checks events under /var/log/journald_boot.log similar to:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
Search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support.

4. Unexpected guest VM reboot

Perform the following steps to identify the scope of the issue:

Write down the names of affected VMs and the time or reboot.Make sure that the VM reboot was not caused by the host reboot by following the steps in the Unexpected host reboot section above.Make sure that all hosts are accessible over the network. If not, then perform the steps described in the Unreachable host section above.Verify that guest OS is supported as per Compatibility Matrix.Run cluster_services_status NCC check to make sure that services on CVM are in a stable state:
	nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
	Check logs inside guest VM as unexpected restart may be a result of issues with the guest OS itself. Log names can be different on different OSes. On Windows VMs, check System and Application event logs. On Linux VMs, check /var/log/messages and messages.<number>.gz files for any critical events preceding host reboot. Consider engaging an OS vendor if you see evidence of issues inside guest OS.Check QEMU logs on the AHV host and compare the signature to find how reboot was triggered:
	Get the VM UUID:
		Prism UI: VM details in VM pageNCLI:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
			
		Go to the host where the VM was running and check the log file in /var/log/libvirt/qemu/<vm-uuid>.log
		It is possible to find where the VM was running by looking on which host log files are present and checking the last lines to find out the most recent log:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
			The output will be displayed for every host where the VM ran, but only one of them will have the most recent date:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
			
		
		Check for the following signature in /var/log/libvirt/qemu/<vm-uuid>.log

		Power OperationLog Snippet"shutdown -P now" in Guest
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					
					"shutdown -r now" in Guest or

					NMI (Kernel panic) - check "NTNX.serial.out.0" log
					
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					
					"virsh shutdown <VM>" or

					VM power operation in Prism
					
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh reboot <VM>" or
					VM power operation in Prism
					
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh reset <VM>" or
					VM power operation in Prism
					
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh destroy <VM>" or
					VM power operation in Prism
					
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					
		
	

If all factors mentioned above are excluded, consider checking /home/data/logs/acropolis.out on Acropolis leader and /var/log/libvirt/qemu/<vm uuid>.log on AHV host where VM was running before restart.

5. Inability to perform VM management actions

In most cases, the inability to perform VM management actions from Prism or acli on one or more AHV hosts is caused by issues with the Acropolis service itself. Start with checking /home/data/logs/acropolis.out on the host where affected VM is located and on Acropolis leader. Look for the "Traceback (most recent call last)" text in the log to identify the start of the stack trace, which indicates that the Acropolis service is not healthy.

Search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support.

6. Stuck VM migration

In most scenarios, VM migration is not stuck, but processing very slowly. Refer to KB 3810 for additional details.

7. Failed VM live migration due to Jumbo frames misconfiguration.

If the Jumbo frames are used, but the physical network switches do not have the correct configuration to allow packets with higher MTU size, the live migration of VMs will fail. Refer to KB 10180 for more information.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.