### Human: Can you tell about Space Accounting - Prism show cluster/container space usage different than total usage from all VMs on it.. ### Assistant: This article explains why space usage on the cluster/container level can be different from total disk usage from all User VMs.. For other Space Accounting issues/questions that are not covered on this page, please take a look at the Space Accounting | Nutanix Storage Usage Guide and FAQ article.When comparing space usage on the cluster/container level with space usage reported in total for all User VMs from this cluster/container, it often can be found these numbers do not match.

Scenario A: comparing cluster-level usage with usage from all User VMs.

Prism UI, VM view table shows all User VMs on the cluster with storage used/allocated for each VM.

It is possible to take the current usage numbers from all User VMs on the cluster and find the total usage number. But total usage from all User VMs almost always will not match with current usage on the cluster (on the storage pool level).

Here is an example, from a lab cluster with only 6 User VMs.One VM has only 1.61 GiB of usage (only guest OS installed), and the remaining 5 VMs have 196.69 GiB of storage usage (usage was created by independently generating some data on the Guest OS level):



In total, all 6 VMs have storage usage of ~986.66 GiB of logical usage, which for an RF2 container should translate into 1.93 TiB of physical usage on cluster/pool level.

At the same time, on the storage pool level (cluster-wide), usage is at ~1.16 TiB of physical usage.

nutanix@cvm$ ncli sp ls | tail -3

This is significantly less than expected for usage on the VMs/VGs level.

Scenario B: comparing container level space usage with usage from all User VMs.

It is possible to compare usage on the container level with total usage from all User VMs/VGs on this container.

Example from the same cluster as in scenario A, where we have a set of Data VMs with an identical amount of data (~196 GiB) written on Guest OS level.Here, usage for a highlighted container named “ctr1_compression” is 25 GiB. But at the same time, on the “Breakdown” view for this container, we can see there is only one VM with two disks and total usage of ~196 GiB.



If a container ha space-saving (data reduction) features enabled, then actual usage on the container level will be noticeably lower than the logical usage on the VM level. 

Details on why total logical usage from all User VMs/VGs does not always match with cluster/container usage numbers.Described comparison scenarios will often show differences in space usage on the cluster/container level compared to what reported usage on the VMs/VGs level (in total).This difference can be explained by the fact that logical usage on VM/VG level is not taking into account several factors:  Space savings (or Data Reduction), Snapshots, Clones, and system garbage.The table below has some more details that explain how each factor can affect space usage differences reported on different levels (cluster/container vs. VMs/VGs).

Factor to take into account.Details on how this factor impacts space usage on Cluster/Container level vs. VM/VG level.Space savings (or data reduction) features are enabled on containers.
			Savings from Compression, Deduplication, and Erasure Coding are only considered on the cluster/container level but not on VM/VG logical level.			Container logical usage reported in Prism is obtained from physical usage related to this container, divided by the replication factor of that container.			The logical usage on a VM/VG level show usage (amount of written data) on virtual disks without considering replication factor or space-saving techniques, or the real amount of data stored on the physical disks in nodes.

			So, due to savings, you can see less usage reported on the cluster/container level compared with VM/VG level.
			Snapshots			(any snapshots on the Nutanix storage layer).
			The usage form snapshots are not reported/reflected in the main VM/VG view or on the container breakdown view in the Prism Element UI. 			But snapshot usage is part of cluster/container usage.			(any snapshots on the Nutanix storage layer).

			Due to snapshots-related usage, you can see cluster/container usage numbers are higher than total VM/VG usage numbers (multiplied by RF factor for case of comparison with storage pool usage).
			Clones
			The Cloned VM/VG will report the same logical usage as the original VM, but the shared amount of data is stored only once and not multiplied on the physical layer as a result of each cloning operation.

			So, due to shared usage from cloning – usage numbers on Cluster/Container level will be less than you can get in total on the VM/VG logical layer.

			Example:			One VM with 1 TiB usage was cloned 5 times. So, in Prism on VM view (or in container breakdown view), you will see 6 VMs, each with 1 TiB logical usage. But on the container level, you will see logical usage of 1 TiB (initial usage of the original VM). And only if/when each cloned VM completely overwrites shared data (so nothing is shared) – container logical usage will match with total VM logical usage (if no other factors are in play).
			System garbage/overhead.    			There are different types of garbage can exist in a cluster.						When some data is deleted from the cluster/container (it can be a whole VM, VM disk, snapshot, or it is just some data/files deleted by a user inside Guest OS) – space is not immediately returned as free. Nutanix cluster usually needs some time to reclaim deleted data and return it as free space. The time required for the reclamation depends on the amount of data to be reclaimed and which data reduction techniques are in use.			And all deleted but not reclaimed yet data is considered garbage.						Also, even if there were no manual actions taken by the user to remove something from the cluster (like VMs/Disks/Snapshots removed, or some data deleted from the Guest OS layer) – the Nutanix cluster is still expected to have some amount of garbage generated due to data overwrites which happens normally during UVMs/Applications work (it can be generated also due to old DR snapshots automatically expiring).						The amount of garbage is heavily dependent on workload patterns in User VMs and on storage container configuration. On heavily loaded clusters the amount of garbage often can reach 5-10% of total usage, and it usually fluctuates in accordance with workload changes (depending on the time of the day for example) and with automatic cleanup/reclamation that happens on a cluster periodically (driven by Curator service).						Garbage usage is only accounted for on a Cluster/Container usage layer, and it is not considered on the VM/VG logical usage layer.						So, due to some amount of garbage present in any Cluster, it is expected to see usage on Cluster/Container level to be higher than expected VM/VG logical usage (if there are no other factors in play here).

Note: Remember that all these factors are typically in play simultaneously (if all of them are in use/enabled), and often they can compensate for each other. For example, additional usage from snapshots can be partially/fully compensated by savings from Erasure Coding. Ultimately, logical usage on the container will match total logical usage from VMs/VGs.. Main recommendation: total logical usage from VM/VG level should not be directly compared with usage on cluster/container level without considering all factors mentioned in the description section of this KB.If you believe that cluster/container usage is not correct for the amount of data in User VMs/VGs - then each mentioned factor should be deeply analyzed to understand where the difference in usage comes from.It is recommended to start the analysis process by checking:

The Storage Overview view in Prism Element to see the data reduction ratio, overall efficiency ratio, amount of savings, and other details.Prism Central - your cluster in question - Capacity Tab: Storage View - this can give a quick overview of current and historical live/snapshot/garbage usage numbers on a cluster or container level.KB Space Accounting | Are the space-saving settings on my containers configured correctly - to see best practices and how to check your current configuration and the amount of savings.KB Space Accounting | Identifying Snapshots on a Nutanix Cluster to learn how to identify snapshots in a cluster and find their reclaimable space.More details about the implementation of different data reduction and data avoidance techniques can be found in the Data Efficiency tech note.

For other Space Accounting issues/questions that are not covered on this page, take a look at the Space Accounting | Nutanix Storage Usage Guide and FAQ article.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.