### Human: Can you tell about NCC Health Check: host_nic_error_check. ### Assistant: The NCC health check host_nic_error_check checks whether the NICs on the host are experiencing errors.. The NCC health check host_nic_error_check checks whether the NICs on the host are experiencing errors. It checks if each NIC has fewer than 1% errors during a span of execution of 1 hour (3600 seconds).

Running the NCC Check

You can run this check as part of the complete NCC health checks:

nutanix@cvm$ ncc health_checks run_all

Or you can run this check individually:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

Sample Output

For status: PASS

nutanix@cvm$ ncc health_checks run_all

For status: FAIL

nutanix@cvm$ ncc health_checks run_all

nutanix@cvm$ ncc health_checks run_all

For status: WARN (changed in NCC-4.6.2)

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID6011DescriptionChecks whether the NICs on the hypervisor are experiencing errors.Causes of failureNIC may be failing.ResolutionsCheck NIC health; if required, replace the NIC as soon as possible. Refer to the Nutanix documentation for instructions.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC packet receive an error rate highAlert Message
			nic_errors packet receive errors on NIC nic_name out of nic_packets in the last {period} seconds
			ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.
Note: Check_id 6011 does not generate an alert when the check fails prior to NCC-4.6.2 
Check ID6013DescriptionChecks whether the NICs on the hypervisor are experiencing rx_crc errors.Causes of failureNIC or its peripherals may be failing.ResolutionsReplace the NIC, cable, SFP+ adapter, or switch port. Refer to the Nutanix documentation for instructions.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC RX_CRC Error Rate HighAlert MessageNIC nic_name in host host_ip has encountered many error_type.ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.

Check ID6014DescriptionChecks whether the NICs on the hypervisor are experiencing rx_missed errors.Causes of failureRX buffer size or transient condition.ResolutionsRX buffer size might be too small to absorb current inbound traffic. This can be a transient condition. If the alert reappears frequently, contact Nutanix support.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC RX_Missed Error Rate HighAlert MessageNIC nic_name in host host_ip has encountered many error_type.ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.

This hardware-related check executes on the below hardware:

Nutanix NXDell XCHPELenovo HXNutanix SXFujitsu XFLenovo SRPowerEdgeIntelIBMHyveVoyager TDC. Ensure that you are running the latest version of NCC when investigating NIC errors, whether you have found them through NCC or Health alerts in Prism. For information on upgrading NCC, refer to KB-2871.

Note: ixgben version 1.7.1 is the minimum version required. If required, the drivers can be upgraded to ixgben version 1.7.17, the latest qualified driver.

What is a CRC error?The sending host computes a cyclic redundancy check (CRC) of the entire Ethernet frame and puts this value in the Ethernet frame's FCS (frame check sequence) section after the user payload. The intermediate switch and the destination host check this computed value against the value they compute to determine if the frame has been corrupted in transit.

Possible causes of NIC errors:

Link flap (interface continually goes up and down)Cable disconnect/connectFaulty external switch portMisconfiguration of the external switch portFaulty NIC portFaulty cableFaulty SFP+ module

Faulty cables and/or SFP+ modules are the most common cause of these errors. If the problem occurs regularly on a particular interface, you should perform testing in a controlled fashion during a change window to isolate the faulty component by swapping cables, modules, and switch ports as needed to isolate the issue.

How do I determine if the Nutanix node is experiencing NIC errors?

Log in to the hypervisor and use the ethtool command to retrieve the RX counts of the NIC interface reported in the alert. Continue checking the counters to determine if they are increasing.

ESXi

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	Or:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Depending on the driver version, the output of "ethtool -S" may differ. Some old versions do not have details like the above. In this case, you can try the command below (ESXi only):

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	We want to avoid any CRC errors, which could greatly impact usability. Collect all errors on all node interfaces and from the upstream switch.

	For each node, run the following command:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	As per the upstream switch, it depends on the vendor. For most, the command "show interface <interface>" should work from a privileged console.

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Look for the following output:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Note: For Mellanox, use:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	AHV
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Depending on the driver version, the output of "ethtool -S" may differ. Some old versions do not have details like the above. In this case, you can try this command:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	For enhanced usage of ethtool, you can incorporate additional counters to effectively detect network issues   

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	By running this command, you can retrieve information about various counters related to network errors, faults, checksums, and frame integrity. This can assist you in identifying potential network issues that require further investigation or troubleshooting.Hyper-V
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	

The two big error counters are rx_crc_errors and rx_over_errors (and, in conjunction, rx_missed_errors/rx_fifo_errors).

rx_crc_errors are caused either by faults in layer 1 (in the past, we have seen failed twinax cables and incorrect types of fibre being used), or issues with jumbo frames on the network. In an environment with 10 Gig switches that use cut-through forwarding (Cisco Nexus, Arista, and Cisco devices using IOS default to Store and Forward switching), any packets that come into the switch will get forwarded out the destination interface once the switch has read the destination MAC address. If that packet has an MTU over what is configured on the interface, it will cut off the packet at the designated MTU, causing the server to receive a malformed packet, which will throw a CRC error.

rx_over_errors are caused when the hardware receive buffer on the physical NIC is full, and some of the received packets have to be dropped at the physical NIC layer. In most cases, the values reported by this counter will equal rx_missed_errors and rx_fifo_errors. The packet drops can happen during high bursts of traffic.Note: rx_crc_errors_phy is the counter used on AHV for some Mellanox cards to count CRC errors. From NCC-4.3.0 onwards, the counter is added to the check.  For NCC versions prior to 4.3.0, this counter should be checked manually using the steps mentioned in the solution section of this KB article.

Should I be concerned that my NIC port shows any NIC errors?As indicated above, NIC errors can be triggered by several scenarios and can usually be ignored. However, continuously increasing NIC errors typically point to a failing physical layer component. The Nutanix alert is triggered when the error rate is considered excessive. This alert should be investigated whenever it is raised, and the source should be corrected.

How does the packet error handling between cut-through and traditional store-and-forward switches differ?Traditional store-and-forward switches store each Ethernet frame and perform various checks such as valid frame length and checksum check. If the frame is valid, it is switched to the destination port. In the succeeding example, if the non-Nutanix node generates a corrupted Ethernet frame, switch port 3 drops it. This behavior differs with cut-through (also known as low latency) switches, which do not store the entire frame before forwarding it. The switch usually only needs to look at the first 12 bytes of a packet to decide. The downside of cut-through processing is that a corrupted frame is forwarded to the destination. Cut-through forwarding relies on the destination (host NIC) to perform the error check and discarding, if necessary.

Example setup for scenarios 1 and 2 below:

Nutanix node A ------- NIC port 0 -------- Switchport 1Nutanix node B ------- NIC port 0 -------- Switchport 2Non-Nutanix node --- NIC port 0 -------- Switchport 3

The following two scenarios are the most common for NIC errors.

Scenario 1. NIC errors are only increasing on node A. Nutanix Node B is not experiencing increasing NIC errors.

Perform basic physical layer checks between Nutanix node A and switch port 1.
	Make sure the cables are properly connected.Check cables for physical damage, pinched fibre, and bend radius issues.If the cable is Twinax, make sure that it is supported. Nutanix ships lengths of 1, 3, and 5 meters Twinax cables. 7-meter cables might work, but the signal can be marginal and varies between switch vendors. Some switch vendors may be able to drive a signal 7 meters while others may be marginal. Use a 5-meter cable if in doubt.If the cable is optical, ensure the proper optics are used. Currently, we only support multimode short-reach optics (850 nm), and 1310 nm optics are not supported.
	Determine if the switch port causes the NIC errors.	If the NIC error is incrementing on Nutanix node A, check switch port 1 for tx output or similar transmission errors. Transmit output errors are usually a good indication that the switch port is the source of the NIC errors.Further isolation of the NIC errors.	If the external switch port does not show any NIC TX errors, then the physical layer media (switch port transceiver, cable) could be causing the NIC RX errors. Proceed if the NIC RX errors continue to increase.
	If using Twinax 10GE cabling, replace the cable. Proceed to step 3d if the NIC errors continue to increase.If using optical links, start by replacing the multimode cable. Proceed to the next step if the RX CRC errors continue to increase.If using optical links, replace the optical transceiver associated with the switch port. Proceed to the next step if the RX CRC errors continue to increase.Reboot the host.Replace the host NIC.
	

Scenario 2. NIC errors are increasing on all Nutanix nodes at the same time.

This scenario usually indicates the use of a cut-through Ethernet switch being used. These switches are unable to perform frame discarding at the switch input port. The first step in troubleshooting this scenario is to confirm that the cut-through switch is being used.	Check the external cut-through switch for incrementing error counters. Typically, a single switch port logs RX input errors, and the remaining ports report TX output errors. In this case, the port reporting RX input errors is attached to the node generating the illegal (CRC, length) frames.show tech from the switch or show interface for all the enabled interfaces on the switch. Look for any interfaces that are flagging RX errors.Run ethtool every minute for 5 minutes. Take rx_crc rate of increase of errors on the Nutanix interfaces.Note the Nutanix nodes that are not flagging these errors. This is to find out where the good nodes are connected to. A network diagram would be helpful.

General Troubleshooting

Remediation Steps for CRC Errors:

There are a few different ways of looking at things.If you have a layer 1 issue, you will see rx_crc_errors, not on every but some nodes.
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	This points to this node sending out malformed packets, and we should see a related CRC error on its upstream switch port. This is because the packets coming out of this host will never be sent back to the port from which they were sent.
	If you see many rx_crc_errors on all nodes, verify that there is no "noisy neighbor" on the network.This could be that a host is broadcasting or multicasting out traffic at a larger MTU size. This process will require more network troubleshooting that may fall outside of Nutanix Support.

The general rule is that the rx_over_errors should be 1 in every 5,000 rx_packets at the most. Use the following command on an ESXi host to determine the total number of packets and the number of rx_over_packets:

[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

Remediation Steps for Missed Errors:

Check the current NCC version if you do not see any performance impact. Make sure that it is upgraded to the latest compatible version. Previous versions of NCC were aggressive in reporting the NIC error rates.If there is a need for flow control, enabling the Ethernet flow control feature will allow the physical NIC to signal the switch to slow down the packet transmission rate.More information about this feature can be found in Wiki - Ethernet flow control.

Flow control should only be enabled if it falls within the SOP for your network environment and if a considerable amount of congestion is encountered on the upstream switch. The impact on the network is as follows:

Flow control operates at a lower layer than TCP or IP and thus is independent of them. In other words, flow control can be used regardless of the higher-level protocols. An important side-effect is that neither TCP nor IP knows what Ethernet's flow control is doing. They operate assuming that there is no flow control other than what they may or may not provide themselves.Flow control functions between two directly connected network devices and flow control frames are never forwarded between links. Thus, two computers connected via a switch will never send pause frames to each other but could send pause frames to the switch itself (and vice versa: the switch can send pause frames to the two computers).Pause frames have a limited duration; they will automatically "expire" after a certain time. The expiration time is set by the device that transmits the pause frame.A paused link is not a discriminator of protocols; it will prevent data from passing across the link other than more pause frames.Another solution is to consider enabling load balancing at the network layer so both interfaces are used and also have additional buffer space. Check KB-5135 for details on how to configure AHV.

Receive Length Error

What is rx_length_error?Each error captured by a driver is explained in Intel 82599 datasheet.

Number of packets with receive length errors. A length error occurs if an incoming packet length field in the MAC header doesn't match the packet length. To enable the receive length error count, the HLREG.RXLNGTHERREN bit needs to be set to 1b. This register counts all packets received, regardless of L2 filtering and receiving enablement.

For rx_length_error, in most cases, the switch port sends BPDU packets without changing the length value properly.Use the two options below to overcome the error:

Portfast
	IOS set STP portfast:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	NXOS set port type edge:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	
	Enable BPDU guard and BPDU filter (in Cisco/Arista switch).

	Highly recommended global BPDU Filter/Guard

	IOS:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	NXOS:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Enable BPDU Filter/Guard on a port:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	HP Switches (Verify the HP Switch Model before running the below commands):

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Also, consider disabling Jumbo Frames on the Switch ports the Nutanix Nodes are connected to (HP switches can come with Jumbo Frames enabled by default). According to the AHV Networking Best Practice Guide, enabling jumbo frames on AHV is not recommended.
	

Note: If you see this alert for a vUSB device, upgrade NCC to the latest version to fix the false issue.
If the steps mentioned above do not resolve the issue, consider engaging Nutanix Support for further investigation. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.