### Human: Can you tell about NCC Health Check: notifications_dropped_check. ### Assistant: The NCC Health Check notifications_dropped_check identifies if any alert notifications from any system components have been dropped by Cluster Health and/or Alert Manager due to various possible downstream service or configuration issues. The check aims to proactively expose when alerts can not be delivered successfully and thus a genuine issue on the cluster that requires attention may otherwise go unnoticed.. The NCC Health Check notifications_dropped_check identifies if any alert notifications from any system components have been dropped by Cluster Health and/or Alert Manager due to various possible downstream service or configuration issues within the last 24 hours. The check aims to proactively expose when alerts can not be delivered successfully and thus, a genuine problem on the cluster that requires attention may otherwise go unnoticed.When the queue of alert notifications waiting to be delivered becomes >= 1000, it may suggest that services such as cluster_health and alert_manager cannot process and deliver the alerts, preventing emails, SNMP traps, and Rsyslog messages from being sent. Notifications that arrive when the queue depth is already at 1000 are dropped, and an email notification is sent to the cluster's configured alert email address(es) to inform that alerts and notifications were dropped.

Two reasons why notifications might be dropping : 

Alert Manager service is down.The RPC queue is full.  

Running the NCC CheckRun this check as part of the complete NCC Health Checks.

nutanix@CVM:~$ ncc health_checks run_all

Or run this check separately:

nutanix@CVM:~$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check does not generate an alert.

Sample outputFor Status: PASS

nutanix@CVM:~$ ncc health_checks run_all

For Status: FAIL

nutanix@CVM:~$ ncc health_checks run_all

For Status: INFO

nutanix@CVM:~$ ncc health_checks run_all

Or:

nutanix@CVM:~$ ncc health_checks run_all

Example email:

nutanix@CVM:~$ ncc health_checks run_all

Output messaging

Check ID111052DescriptionSome alert notifications were droppedCauses of failureOne or more cluster services are not healthy.ResolutionsContact Nutanix Support.ScheduleThis check is scheduled to run every day by default.
Note: This NCC check has multiple enhancements in NCC v4.1.0 - Please upgrade your cluster to NCC v4.1.0 or above and re-run the check.. If this alert is seen and/or emails are received, ensure the NCC is up to date and running the latest version (at least NCC-3.9.2.1 or higher). Due to an upstream change in how alert retries are handled and logged, this health check is over-sensitive in NCC 3.9.0/3.9.1 and may be triggered unnecessarily. After the upgrade, re-run the check and confirm the PASS result. If you continue to see a non-PASS result, you may have a genuine issue, or the now-stale trigger in the JSON state file needs to be cleaned up (per step #5 below.) Use the following steps to confirm and resolve.Check the Alert page in the Prism web console to verify what alert events are accounted for.
	If you see there are no events/notifications, but the check still fails, check :

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json

	If we look at the .json file, we can see that timestamps for dropped notifications increment even though there are no alerts in Prism. This is because we keep retrying to send stuck notifications:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Investigate why notifications are being dropped.
	One of the reasons could be that you might be using "admin" account to access some APIs. You should create a separate user with admin privileges instead. Refer to 5.11 Security Guide - Creating A User Account​​​Confirm all CVMs and Cluster Services are up and stable:
		nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
		Validate that the configured SMTP server is reachable and can relay the alert notification emails from the cluster's configured source email address to the destination email address(es).Validate the configured SMTP server security settings.
	Check older health_server.logs also to make sure the error has not rolled off:
	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	The notifications_dropped_check health check might still complain about the same issue based on data on a CVM which was previously but not currently the Cluster Health service leader, as it is looking for any indication of issues by way of a non-zero value in the JSON state file, despite current Cluster Health leadership. Drops might increment only on the leader, so we have to clear the stats on the old leader where the problem may have previously occurred.	SCP the file from the CVM where the values are 0 and replace it. To replace this for all CVMs, you can use this script:
	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Example:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Run the NCC health check again and ensure the check results in a PASS status.

If the above solution does not help you or none of the CVMs notifications_dropped_state.json file has the value of 0 to scp, collect the following information, and consider engaging Nutanix Support.


	An output of the following NCC health check:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	


	List of alerts:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	


	An output of the following command from the PE cluster or the PC VM depending on where the alert was received:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.