### Human: Can you tell about HDD or SSD disk troubleshooting. ### Assistant: This article describes how to troubleshoot a failed disk and identify which component is causing the failure. If you have received an alert for a failed disk, you can troubleshoot a specific node instead of examining an entire cluster.. When a drive is experiencing recoverable errors, warnings or a complete failure, the Stargate service marks the disk as offline. If the disk is detected to be offline 3 times within the hour, it is removed from the cluster automatically and an alert is generated (KB 4158 or KB 6287).

If an alert is generated in Prism, the disk must be replaced. Troubleshooting steps do not need to be performed.

If a failed disk is encountered in a Nutanix Clusters on AWS, once the disk is confirmed to have failed proceed to condemn the respective node. Condemning the affected node will replace it with a new bare metal instance of the same type.. Once the disk is replaced, an NCC health check should be performed to ensure optimal cluster health. However, if an alert is not generated or further analysis is required, the steps below can be used to troubleshoot further.Before you begin troubleshooting, verify the type of HBA controller.Caution: Using the SAS3IRCU command against an LSI 3408 or higher HBA can cause NMI events that could lead to storage unavailability. Confirm the HBA controller before using these commands. To determine what type of HBA is used, look for the controller name located in /etc/nutanix/hardware_config.json on the CVM.Example of the output when SAS3008 is used.  In this case, the command SAS3IRCU is the correct command to use. Note the "led_address": "sas3ircu:0,1:0" line:

"node": {

Example of the output when SAS3400/3800 (or newer) is used. In this case, using SAS3IRCU would be ill-advised. Use the storcli command instead. For information on StorCLI refer to KB-10951. Note "led_address": "storcli:0" line.

"node": {

 

Identify the problematic disks

	Check the Prism Web console for the failed disk. In the Diagram view, you can see red or grey for the missing disk.Check the alerts in the Prism Web console for the disk alerts, or use the following command to check for disks that generate the failure messages.
		"node": {
		Check if any nodes are missing mounted disks. The two outputs should match numerically.
		Check the disks that are mounted on the CVM (Controller VM).
			"node": {
			Check the disks that are physical in the CVM.
			"node": {
			Check if the status of the disks is all Online and indicated as Normal.
			"node": {
			
		Validate the expected number of disks in the cluster.
		"node": {

		The output of the command above should be the sum of the outputs of steps 1c.i and 1c.ii.

		There are instances where the number can be higher or lower than expected. So, it is an important metric that can be compared to the disks listed in step 1b.
		Look for extra or missing disks.
		"node": {
		Check that all disks are indicated as mounted rw (read-write) and not ro (read-only).
		"node": {
		
	Identify the problems with the disks nodes
	Orphaned disk ID
		This is a disk ID that the systems no longer use but was not properly removed. Symptoms include seeing an extra disk ID listed in the output of ncli disk ls.

		To fix the orphaned disk ID:

		"node": {

		Ensure that you validate the disk serial number and that the device is not in the system. Also, ensure that all the disks are populating using lsscsi, mount, df -h, and counting the disks for the full-disk population.
		Failed disk and/or missing disk
		A disk becomes failed and/or is no longer visible to the system for several reasons. There are two approaches to solving a disk problem. The top-down approach is the software representation of the disk down to the hardware/physical layer. The bottom-down approach documents the investigation of a disk from a bottom-up perspective since the 'top' layer can be relative to the problem reported by the software layer (Prism, Hades, Stargate, etc.).

		Check if the disk is visible to the controller as it is the device whose bus the disk resides on.

		lspci - displays the PCI devices seen by the CVM.

			NVME device - Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01).SAS3008 controller - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) - LSI.SAS2008 controller - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03).SAS2308 controller (Dell) - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05).MegaRaid LSI 3108 (Dell) - RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02).LSI SAS3108 (UCS) - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3108 PCI-Express Fusion-MPT SAS-3 (rev 02).
			lsiutil - displays the HBA (Host Bus Adapter) cards perspective of the ports and if the ports are in an UP state. If a port is not up, either the device on it does not respond, or the port or connection to the device is bad. The most likely scenario is the device (disk).
			"node": {
			lsscsi - lists the SCSI bus devices seen that include any HDD or SSD (except NVME, which does not pass through the SATA controller).sas3ircu - reports slot position and disk state. It is useful for missing disks or verifying that disks are in the correct slot. (Do NOT run the following command on Lenovo HX hardware as it may lead to HBA lockups and resets)  
			"node": {
			storcli - reports slot position and disk state. Commonly found on Lenovo Chassis but will be included on NX G8 platforms. 
			Slot # (should match the physical slot)

			State is the state of the disk. New disks will be listed as Available (AVL). After bootup and SCSI inquiry healthy disks will show as Ready (RDY).			In the example below, a disk was detected but was never marked as Ready (RDY). It remains in Available (AVL) status due to a bad disk. In the next section, we see the related mpt3sas dmesg output showing the SCSI inquiries constantly polling this disk.  			(Note that the Serial No has been truncated and replaced by the Firmware Revision LC72 repeated for this platform. This was consistent for the systems in this particular cluster. This does not indicate a problem, simply a unique behaviour.)

			"node": {
			Check the CVM's dmesg for LSI mpt3sas messages. We should typically see one entry for each physical slot. (The below example shows SAS address "0x5000c5007286a3f5" is repeatedly checked due to a bad/failed disk. Note how the other addresses are detected once, and the suspect is repeatedly being polled.)
			"node": {
			smartctl - if Hades indicate that a disk is checked by smartctl 3 times in an hour, it is automatically failed.
			"node": {

			See KB 8094 for troubleshooting with smartctl.
			Check for offline disks using NCC check disk_online_check.
			"node": {

			See KB 1536 for further troubleshooting offline disks.
			Check if NCC has any errors indicated for disks, mount paths being offline, or failed disks in the system.Confirm if disks are seen from LSI Config Utility:
			Enable LSI HBA OPROMGo to “Advanced-->PCIe/PCI/PnP Configuration”, change CPU1 SLOT2 PCI-E 3.0 x8 OPRM to “Enabled” as below showing,Press "F4"Enter LSI Config Utility on bootup:
				
				Press <Enter> twice to confirm if disks are seen or not:
				

				
				
			There can be a case where the disk is DOWN in lsiutil, usually after a replacement or an upgrade of the disks. When all the above checks are carried out, and the disk is still not visible, compare the old and new disk "disk caddy or tray". Ensure the type is the same. There can be cases where an incorrect disk type is dispatched, and it does not seat properly in the disk bay hence not being detected by the controller.
		
	

 

Identify the node type or the problematic node

	Run ncli host ls and find the matching node ID.

	Specific node slot location, node serial, and node type is important information to document in case of recurring issues. It also helps to track the field issues with the HBA's, node locations, and node types.
	Identify the failure occurrence
	Check the Stargate log.
		The stargate.INFO log for the corresponding period indicates if Stargate saw an issue with a disk and sent it to the Disk Manager (Hades) to be checked or had other errors accessing the disk. Use the disk ID number and serial number to grep for in the Stargate log on the corresponding node the disk is in.
		The Hades log contains information about the disks it sees and the health of the disks. It also checks which disk is metadata or Curator disk and selects one if one did not already exist in the system or was removed/disappeared from the system. Check the Hades log.Check df -h in /home/nutanix/data/logs/sysstats/df.INFO to see when the disk was last seen as mounted.Check /home/nutanix/data/logs/sysstats/iostat.INFO to see when the device was last seen.Check /home/log/messages for errors on the device, specifically using the device name, for example, sda or sdc.Check dmesg for errors on the controller or device. Run dmesg | less for the current messages in the ring, or look at the logged dmesg output in /var/log.
	Identify the reasons for disks failure
	Check when the CVM was last started if the disk's last usage data were not available. Again, reference the Stargate and the Hades logs.Check the Stargate log around the time of disk failure. Stargate sends a disk to Hades to check if it does not respond in a given time and ops timeout against that disk. Different errors and versions represent it differently, so always search by disk ID and disk serial.
	Check the count of disk failure
	If a drive failed more than once in this slot and the disk was replaced, it would indicate a potential chassis issue at that point.
	Check if lsiutil is showing errors.
	If lsiutil shows errors evenly on multiple slots, it indicates a bad controller.
	

Sample command output for reference:

"node": {

On a Nutanix Clusters on AWS:

"node": {

To run the commands in a Hyper-V environment, use winsh on the CVM:

"node": {

"node": {

NOTE: For ID: 191, G-Sense_Error_Rate for Seagate HDD's can be safely ignored unless there is performance degradation. G-Sense_Error_Rate value only indicates HDD adapting to shock or vibration detection. Seagate recommends not to trust these values as this counter dynamically changes the threshold during runtime.

"node": {. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.