### Human: Can you tell about AOS Only - What to do when /home partition or /home/nutanix directory on a Controller VM (CVM) is full. ### Assistant: This article describes ways to safely free up space if /home partition or /home/nutanix directories of a CVM become full or do not contain sufficient free space to facilitate an AOS upgrade.. Important Notes:

Do not use this KB or the cleanup script if /home partition is exceeding the limit on a Prism Central VM (PCVM). For the PCVM issue, refer to KB 5228.Login to the CVMs as nutanix userDo not treat the Nutanix CVM (Controller VM) as a normal Linux machine.Do not use rm -rf under any circumstances unless stated. It will lead to data loss scenarios.If you are running LCM-2.6 or LCM-2.6.0.1, LCM log collection fills up /home directory please refer KB 14671 for workaround.If you receive /home partition usage high alert on a cluster running NCC 4.0.0, also check KB 10530.You can review the specific clusters affected by this alert via the discoveries on the Support Portal powered by Nutanix Insights hereContact Nutanix Support if you have any doubts.

CVM /home partition or /home/nutanix directory being full can be reported in two scenarios:

The NCC health check disk_usage_check reports that the /home partition usage is above the threshold (by default, 75%).The pre-upgrade check test_nutanix_partition_space checks if all nodes have a minimum of 5.6 GB space in the /home/nutanix directory.

The following error messages are generated in Prism by the test_nutanix_partition_space pre-upgrade check:

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB

Nutanix reserves space on the SSD tier of each CVM for its files and directories. These files and directories are in the /home folder you see when you log in to a CVM. The size of the /home folder is capped at 40 GB so that most of the space on SSD is available for user data.

Due to the limited size of the /home partition, running low on free space and triggering Prism Alerts, NCC Health Check failures or warnings, or Pre-Upgrade Check failures is possible. These guardrails exist to prevent /home from becoming full, as this causes data processing services like Stargate to become unresponsive. Clusters where multiple CVMs' /home partitions are 100% full often result in downtime of user VMs.

The Scavenger service running on each CVM is responsible for the automated clean-up of old logs in /home. Improvements to the Scavenger service scope were made in AOS 5.5.9, 5.10.1, and later releases. For customers running earlier AOS releases, or in special circumstances, it may be necessary to manually clean up files in some directories to bring the space usage of /home down to a level that will permit the AOS upgrade.

When cleaning up unused binaries and old logs on a CVM, it is important to note that all the user data partitions on each drive associated with a given node are also mounted within /home. Nutanix advises strongly against using undocumented commands like rm -rf /home since this will also wipe the user data directories mounted within this path. This article aims to guide you through identifying the files causing the CVM to have low free space and removing only those that can be deleted safely. . General Guidance

Checking the space usage in /home.

	To accommodate a potential AOS upgrade, usage should be below 70%. Use the df -h command to verify the amount of free space in /home. In the example below, CVM x.x.x.12 /home usage is 81%.

	nutanix@cvm$ allssh "df -h /home"

	CVM /home partition information can be collected using the logbay command (NCC 4.0.0 and above, KB 6691).

	nutanix@cvm$ allssh "df -h /home"
	Cleaning unnecessary files under /home directory.
	If you have any open cases with pending Root Cause Analysis, check with the case owner whether these log files are still required or can be discarded.

	Warnings:

	DO NOT treat the Nutanix CVM (Controller VM) as a normal Linux machine.DO NOT use rm -rf under any circumstances unless stated. It will lead to data loss scenarios.Contact Nutanix Support if you have any doubts.
	

Method 1: Using approved script

Download and run KB-1540_clean_v10.sh to clean files from approved directories. Note: This script is NOT qualified to be used on Prism Central VM.

From any CVM, run the following commands to download KB-1540_clean_v10.sh:

	nutanix@cvm$ allssh "df -h /home"
	Deploy the script to a local CVM or all CVMs of the cluster:
	nutanix@cvm$ allssh "df -h /home"
	Execute the script to clear files from approved directories.
	Help
		nutanix@cvm$ allssh "df -h /home"
		Interactive mode
		nutanix@cvm$ allssh "df -h /home"
		Non-interactive mode
		nutanix@cvm$ allssh "df -h /home"
		
		Note: If the output of the script or its coloring looked incorrect, try to set the environment variable before running the script, or use "--no_color" option:

	nutanix@cvm$ allssh "df -h /home"
		Interactive mode

	Main menu
				All PlansSwitch displaying plans (all / concerned)RescanRescan usages and update tableList AllList all targetted files/directoriesRun AllRun or Dry-run all plansExport AllExport all targetted files/directories to CSV##: Plan #Go to the plan item menu
				Plan item menu
				RescanRescan and update tableOperationChange operation (remove/shelter/etc.)ListList targetted files/directoriesDryrunDry-run this planRunRun this planExportExport targetted files/directories to CSV
				
	Non-interactive mode

	Commands
				$ ./nutanix_home_clean.py --scanScan and show the usages.$ ./nutanix_home_clean.py --list							$ ./nutanix_home_clean.py --list=<##>List up the target files for all plans or specific plan-##.$ ./nutanix_home_clean.py --dryrun							$ ./nutanix_home_clean.py --dryrun=<##>Dry-run all plans or specific plan-##.$ ./nutanix_home_clean.py --run							$ ./nutanix_home_clean.py --run=<##>Run all plans or specific plan-##.
				Additional Options

				--operation=<op>Choose operation (remove, shelter)--sheletedir= <path>Set the sheltering location.--yesFor skipping disclaimer and confirmation.
				
	 Cleaning up after the troubleshooting	The downloaded script files, logs and exported files are expected to be removed manually after every troubleshooting. The total size of these files should be small and will not affect CVM's filesystem. You can remove the following files once the script becomes unnecessary.	<yymmdd-hhmmss> is the creation date and time.
	In the CVM where the KB script is deployed (/home/nutanix/tmp/):		KB-1540_clean.sh - downloaded file from the KB		deploytool_yyyymmdd-hhmmss.log - deployment script's log (unnecessary after deployment)		nutanix_home_clean.py - main KB script		nutanix_home_clean_config.py - config file for the main scriptIn the rest of the CVMs in the cluster - if deployed to all CVM in step 2:		nutanix_home_clean.py - main KB script		nutanix_home_clean_config.py - config file for the main scriptEvery CVM where nutanix_home_clean.py is run:		KB-1540_v10_yyyymmdd_hhmmss_nutanix_home_clean.log - KB script's log		KB-1540_v10_yyyymmdd_hhmmss_export_*.csv - exported files (if exported)
	The following command can remove all of the above:

	nutanix@cvm$ allssh "df -h /home"
	

Method 2: Manual method

PLEASE READ: Only the files under the directories stated below are safe to delete. Take note of the specific guidance for removing files from each directory. Do not use any other commands or scripts to remove files. Do not use rm -rf under any circumstances.

Removing old logs and core files. Only delete the files inside the following directories and not the directories themselves.

	/home/nutanix/data/cores//home/nutanix/data/binary_logs//home/nutanix/data/ncc/installer//home/nutanix/data/log_collector//home/nutanix/prism/webapps/console/downloads/NCC-logs-*

	Use the following syntax for deleting files within each of these directories:

	nutanix@cvm$ allssh "df -h /home"
	Removing old ISOs and software binaries. Only delete the files inside the following directories and not the directories themselves.
	Check the current running AOS version under "Cluster Version":

	nutanix@cvm$ allssh "df -h /home"

	/home/nutanix/software_uncompressed/ - The software_uncompressed folder is only in use when the pre-upgrade is running and should be removed after a successful upgrade. If you see a running cluster that is currently not upgrading, it is safe to remove everything within the software_uncompressed directory. Delete any old versions other than the version to which you are upgrading./home/nutanix/foundation/isos/ - Old ISOs of hypervisors or Phoenix./home/nutanix/foundation/tmp/ - Temporary files that can be deleted.
		Use the following syntax for deleting files within each of these directories:

		nutanix@cvm$ allssh "df -h /home"
		/home/nutanix/software_downloads/
		If the files under the software_downloads directory are not required for any planned upgrades, remove them from Prism Web Console > Settings> Upgrade Software. Also check File Server, Hypervisor, NCC, and Foundation tabs to locate the downloads you may not require. The example below illustrates two versions of AOS available for upgrade, each consumes around 5 GB. Click on the 'X' to delete the files.

		

		Uncheck the “Enable Automatic Download” option if it is checked. Left unmonitored, the cluster will download multiple versions, unnecessarily consuming space in the home directory.
		
	Re-check space usage in /home using df -h (see General Guidance of this article) to confirm that it is now below 70%.

Note: If you cannot delete the files with the below error and space not claimed, contact Nutanix Support for assistance.

nutanix@cvm$ allssh "df -h /home"

Important Notes for NC2 Clusters:It has been observed in some instances of NC2 clusters that /tmp gets close to full. You can follow the below steps to clean ~/tmp directory.

SSH to the affected CVM and check the disk usage by running "df -h" command:
	nutanix@cvm$ allssh "df -h /home"
	In the above output, we can see /tmp is showing 100%. Change the directory to ~/tmp & sort them using 'sudo du -aSxh /tmp/* | sort -h'.
	nutanix@cvm$ allssh "df -h /home"
	From the output you receive above, manually delete files that are bigger than 12K. For example, see below files deleted from the above output.
	nutanix@cvm$ allssh "df -h /home"
	After deleting, you can check available free space using 'df -h':
	nutanix@cvm$ allssh "df -h /home"
	As you can see, available free space now shows 6%. You can further recheck with:
	nutanix@cvm$ allssh "df -h /home"
	or

	nutanix@cvm$ allssh "df -h /home"
	

Even after cleaning up files from the approved directories, /home usage is still above the threshold

Contact Nutanix Support for assistance. Under no circumstances should you remove files from any other directories aside from those recommended by this article, as these may be critical to the CVM performance or may contain user data.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.