### Human: Can you tell about NCC Health Check: pcvm_disk_usage_check. ### Assistant: The NCC health check pcvm_disk_usage_check verifies that the amount of disk space used in the Prism Central VM is within limits.. The NCC health check pcvm_disk_usage_check verifies that the amount of disk or system partition usage in the Prism Central (PC) VM is within limits.

This check has the following parts:


	Checking the individual data disk usage (added in NCC 3.5.1):

	If usage is more than 75% for several hours, a WARNING is returned to identify the disk.If usage is more than 90% for several hours, a FAIL is returned to identify the disk.
	
	Checking the overall data disk usage (added in NCC 3.10.1):

	
		If overall usage is more than 90% for several hours, a WARNING is returned.
		
	
	Checking the Prism Central VM system root partition usage (added in NCC 3.9.4). Returns only FAIL message if the partition usage exceeds 95%.
	
	Checking the Prism Central VM home partition usage (added in NCC 3.9.4):

	If the usage is more than 75%, a WARNING is returned.If the usage is more than 90%, a FAIL is returned.
	
	Checking the Prism Central VM CMSP partition usage (added in NCC 3.10.1):

	If usage is more than 75%, a WARNING is returned.If the usage is more than 90%, a FAIL is returned.
	Checking the Prism Central VM Upgrade disk partition usage (added in NCC 4.6.0):
	If the usage is more than 70%, a FAIL is returned.This check runs every 5 mins.If there are more than 5 failures (30 mins), a critical alert is raised.
	
Please note: If you are running LCM-2.6 or LCM-2.6.0.1, LCM log collection fills up /home directory please refer KB 14671 for workaround.

Running the NCC checkRun the NCC check as part of the complete NCC health checks.

nutanix@pcvm$ ncc health_checks run_all

Or run the pcvm_disk_usage_check check separately.

nutanix@pcvm$ ncc health_checks run_all

You can also run the checks from the Prism Web Console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every 5 minutes, by default.

This check will generate an alert after 5 consecutive failures across scheduled intervals.

Sample OutputsFor status: PASS

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM data disk, e.g. /dev/sdc1)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM overall MultiVDisk)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM data disk, e.g. /dev/sdc1)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on root partition, i.e. /)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM home partition, i.e. /home)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM home partition, i.e. /home)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM CMSP partition, i.e. /dev/sde)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM CMSP partition, i.e. /dev/sde)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM upgrade disk partition, i.e. /home/nutanix/upgrade)

nutanix@pcvm$ ncc health_checks run_all

Check Disk Usage in PC VMFollowing is an example of how to check disk usage on a PC VM.

nutanix@pcvm$ ncc health_checks run_all

Partition for upgrade disk would show up similar to: 

nutanix@pcvm$ ncc health_checks run_all
 

Output messaging

Check ID101059DescriptionCheck that disk space usage on the Prism Central VM is within limits.Causes of failureHigh disk usage in the Prism Central VM.ResolutionsRefer to KB 5228 for further details.ImpactPrism Central VM may run out of storage space to store data.Alert IDA101059Alert TitlePrism Central VM disk usage highAlert MessagePrism Central VM IP disk usage exceeds warning limit x % for disks: disks.			Prism Central VM IP overall MultiVDisk usage exceeds warning limit of  2321329924 KB			Prism Central VM IP disk usage exceeds critical limit x % for disks: disks.

Check ID200316DescriptionChecks if Prism Central system root partition usage is within the threshold to ensure uninterrupted operations.Causes of failureIncreased Prism Central VM system root partition usage due to excessive logging or incomplete maintenance operation.ResolutionsReduce Prism Central VM system root partition usage by removing any known temporary or unneeded files. Refer to KB 5228 for further details.ImpactIf the Prism Central VM system root partition is highly utilised, certain maintenance operations, such as upgrades, may be impacted. If the Prism Central VM system root partition is 100% utilized, services may stop and impact Prism Central cluster management functions.Alert IDA200316Alert TitlePrism Central VM System Root Partition Space Usage HighAlert MessageDisk space usage for root partition mount_path on entity-ip_address has exceeded threshold%.

Check ID200317DescriptionChecks if Prism Central home partition usage is within the threshold to ensure uninterrupted operations.Causes of failureIncreased Prism Central VM home partition usage due to excessive logging or incomplete maintenance operation.ResolutionsReduce Prism Central VM home partition usage by removing any known temporary or unneeded files. Refer to KB 5228 for further details.ImpactIf the Prism Central VM home partition is highly utilised, then certain maintenance operations, such as upgrades, may be impacted. If Prism Central VM home partition is 100% utilised, then services may stop and impact cluster storage availability.Alert IDA200317Alert TitlePrism Central VM home partition disk usage highAlert MessagePrism Central VM IP home partition disk usage exceeds warning limit x %.			Prism Central VM IP home partition disk usage exceeds critical limit x %.

Check ID200328DescriptionCheck that platform disk space usage on the Prism Central VM is within limits.Causes of failureHigh disk usage in the Prism Central VM.ResolutionsRefer to KB 5228 for further details.ImpactPrism Central VM may run out of storage space to store data.Alert IDA200328Alert TitlePrism Central VM platform disk space usage highAlert Smart TitlePrism Central VM svm_ip platform disk space usage highAlert MessagePlatform disk space usage in Prism Central VM svm_ip exceeds percentage_exceed% for disk(s): disk_paths.

Check ID200334DescriptionChecks if Prism Central upgrade disk usage is within the threshold to ensure uninterrupted upgrade operations.Causes of failureIncreased Prism Central VM upgrade disk usage due to the presence of multiple Prism Central Installer filesResolutionsReduce Prism Central VM upgrade partition usage by removing the Prism Central Installer files which are not needed. Refer to KB 5228 for further details.ImpactIf the Prism Central VM upgrade disk is highly utilised, then the Prism Central Upgrade would fail due to lack of space in the upgrade disk.Alert IDA200334Alert TitlePrism Central VM upgrade disk usageAlert MessagePrism Central VM <IP> upgrade disk usage exceeds critical limit x%

Scenarios that trigger pcvm_disk_usage check Warn/Fail on /home partition
Scenario 1The /home directory on long-running PC instances might reach close to its maximum limit as older Prism code is not cleaned up:

nutanix@pcvm$ ncc health_checks run_all

Nutanix Engineering is aware of the issue and is working on a fix in a future release to further clean up older Prism codes. For the workaround, refer to the corresponding solution section.

 

Removing older sysstats logsIf you have checked all the usual places but still need to clean up space, you can get the customer’s permission to remove older sysstats logs. Double-check that there are no open support cases with pending RCAs before proceeding, as this data may not yet have been collected.

nutanix@pcvm$ ncc health_checks run_all

Scenario 2Hyperkube logs (kublet logs) are not being cleaned up after enabling CMSP / microservices on Prism Central. If Cluster Maintenance Utilities (CMU) has been updated to version 2.0.3 using LCM, the included Scavenger version is missing the capability to clean up certain logs related to CMSP microservices. As a result of this issue, users may find that Prism Central services are not starting or that they cannot log in to PC UI. This issue will first fill up the /home/nutanix/data/sys-storage/NFS_.../ directory, after which the kubelet logs will start filling up the root partition in the /tmp folder.

nutanix@pcvm$ ncc health_checks run_all

We may find hyperkube.ntnx* logs getting generated and not rotated correctly when CMSP is enabled. Check if the directory /home/nutanix/data/sys-storage/NFS.../kubelet/ is excessively filled with these logs. 

nutanix@pcvm$ ncc health_checks run_all

When working as intended, we should see less than 10G used in this kubelet folder, and the file count for kubelet* would be less than ~15
We may additionally see similar kubelet logs filling up space in /tmp, after space in the /home/nutanix/data/sys-storage/NFS... directory has been exhausted. 

nutanix@pcvm$ ncc health_checks run_all

Scenario 3Starting PC.2022.6 - a dedicated 30 GB disk is created and mounted for PC upgrades. This Vdisk will be used for downloading and extracting upgrade binaries from consecutive upgrades. If the Prism Central VM upgrade disk is highly utilised, then the Prism Central Upgrade would fail due to lack of space in the upgrade disk.

nutanix@pcvm$ ncc health_checks run_all

Note: In case several services are enabled on Prism Central, such as msp, karbon, calm, flow, and objects, we would see /home usage to be high as each of the services will generate several logs and configuration files. Nutanix engineers are constantly working towards improving the /home usage. If none of the above scenarios matches and this affects Prism Central upgrade, engage Nutanix Support team to help manually cleanup /home by trimming the logs. Scenario 4Due to the log file rotation issue, Adonis logs directory usage is high. This log directory usage should not be over than 1G. For Scale-Out Prism Central deployments, check the file usage on each of the Prism Central VMs:

nutanix@pcvm$ ncc health_checks run_all

Scenario 5In certain cases, catalina.out may consume a large amount of space on the Prism Central VM.

SSH to the Prism Central and check if /home/nutanix/data/logs/catalina.out is consuming a huge amount of space:

nutanix@pcvm$ ncc health_checks run_all. If the check reports a WARN or FAIL status, disk usage is above the threshold and needs investigation. Generally, space utilization can be queried using df -h. The output below shows the mount points as follows:

/dev/sdb1 is root partition/dev/sdb3 is home partition/dev/sdc1 is data disk partition

nutanix@pcvm$ df -h

Data disk usage (/dev/sdXX) or overall multivdisk usage:

Verify the number of VMs supported for the particular Prism Central size is within the limit (consult the Prism Central Guide for your version from the Support Portal for the limits). Contact Nutanix Support. While opening a support case, attach the output of the following commands to the case.

nutanix@pcvm$ df -h

Prism Central VM home partition (/home):

Inspect the NCC output to determine which Prism Central VM has high usage, then perform the following:

Log in to the Prism Central VM.Use the cd command to change the location to the /home partition.List the contents of the directory by size using the command below:
	nutanix@pcvm$ df -h

	Examine the output for any large unused files that can be deleted.
	Run the du command below to list the usage of each file and sub-directory:
	nutanix@pcvm$ df -h

	Examine the output of large sub-directories. You can run the du command for each sub-directory in question to further identify large unused files that can be deleted.
	Below are some common sub-directories of /home where large unused files are likely to exist:
	/home/nutanix/software_downloads/ - delete any old versions other than the versions you are currently upgrading./home/nutanix/software_uncompressed/ - delete any old versions other than the versions you are currently upgrading./home/nutanix/data/cores - delete old stack traces that are no longer needed./home/nutanix/data/log_collector/ - delete old NCC Logs with NCC-logs-2018-07-20-11111111111111-1032057545.tar format./home/nutanix/foundation/isos/ - old ISOs./home/nutanix/foundation/tmp/ - temporary files that can be deleted.
	

If the above steps do not resolve the issue or if the issue matches one of the scenarios presented earlier in this article, follow the solution steps outlined below.
Prism Central VM root system partition (/):Consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

Prism Central VM CMSP partition (/dev/sde):Consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

 

Scenario 1As a workaround, remove the directories that do not have registered clusters with the corresponding version.

Find the AOS versions of all the registered PEs. You can do so from Prism Central > Hardware > Clusters > AOS version column.List the PE apache console directories at /home/apache/www/console/

nutanix@pcvm$ df -h

If any PE apache console directories that do not correspond to registered PE versions are present, they should be safe to clean up.
If you require further assistance with the cleanup, consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

 

Removing older sysstats logsIf you have checked in all the usual places but still need to clean up space, you can get the customer’s permission to remove older sysstats logs. Please double-check that there are no open support cases with pending RCAs before proceeding, as this data may not yet have been collected.

nutanix@pcvm$ df -h

You can use the following command to remove gzipped sysstats logs older than a certain date. In the example below, you will remove sysstats logs from all PCVMs that are older than 3 days (or 4320 minutes).

nutanix@pcvm$ df -h

Scenario 2If your Prism Central instance matches this scenario, refer to KB-12707 Scenario #2 and open a case with Nutanix Support for assistance with in recovering from the issue.Scenario 3The increase in Prism Central VM, upgrade disk usage, is due to multiple Prism Central Installer files. Reduce Prism Central VM upgrade partition usage by removing the Prism Central Installer files that are not needed.This partition /home/nutanix/upgrade is designed to hold upgrade-related files. So, any other file within this directory can be deleted.The partition usage as listed in df -h:

nutanix@pcvm$ df -h
To check the contents of the disk:

nutanix@pcvm$ df -h

To remove the unwanted files, use the below command:

nutanix@pcvm$ df -h
Scenario 4Nutanix is aware of the issue. The fix for this issue will be made available in a future PC release. For a workaround, engage Nutanix Support.

Scenario 5

If you see catalina.out log file is consuming a lot of space, use the following command to restart the prism service on the PCVM. 

nutanix@pcvm$ df -h. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.