### Human: Can you tell about Performance benchmarking with CrystalDiskMark on Nutanix. ### Assistant: Specifics about using the CrystalDiskMark on Nutanix. CrystalDiskMark is an open-source disk drive benchmark tool for Microsoft Windows. Based on Microsoft MIT-licensed diskspd tool, this graphical benchmark is commonly used to test storage performance. It generates a workload with a varying number of queues and threads. The detailed information about using the parent tool diskspd can be found in this Nutanix KB article here. If there is any help needed feel free to open a support case to help you with CrystalDiskMark. . CrystalDiskMark default settingsCrystalDiskMark benchmark is very simple to use and provides a one-click experience for storage benchmarking. However, the default settings are not a summary for testing enterprise-level storage but rather built for PC/laptop local disk testing. Please use the latest version of CrystalDiskMark for testing.Let's take a closer look at the settings of the CrystalDiskMark:Figure 1: Default CrystalDiskMark settings

Working SetBy default, CrystalDiskMark is set to a 1GiB test file size. It is recommended to test with a realistic working set size (the "hot" data which is frequently read and written) of the production application or at least get closer to the size that simulates the real life behaviour. Nutanix as a distributed architecture uses DRAM as read cache as well as Oplog as a write cache for random data and low parallel sequential write patterns. Hence the 1 GiB data will get into Oplog first and later into DRAM while parts of the test data will potentially get read from Oplog. This is unlikely to happen in real life. While customers often compare CrystalDiskMark results on Nutanix with existing SAN based storage systems and local storage solutions, the 1 GiB dataset will come 100% from the SAN cache or Raid Controller cache for both reads and writes and will show how this extremely small dataset with a short run time would behave differently.The below example illustrates the 100% reads from DRAM on a Nutanix based system:Figure 2: Reads with DRAM cachingFor this reason, it is important to understand what the working set would be before doing the benchmarking. Typically, the application owners can provide such information. For spot testing, we recommend using at least a 16 GiB file to bypass the controller cache for both Nutanix and SAN based storage to see realistic synthetic storage performance benchmarking results.

Queues and ThreadsCrystalDiskMark allows setting queues and threads. All four of the default tests are single-threaded, which does not show a realistic I/O profile for the vast majority of workloads as most modern applications are multi-threaded. With Database products like Microsoft SQL, the default of parallel execution is set to a given amount of CPUs configured in the VM. More information about MAXDOP (max degree of parallelism) can be found here. It is highly recommended to understand the application as best as possible and identify whether it uses single-threaded or multi-threaded processing before doing the benchmarking. The test should be as close to the production workload as possible to best understand what the performance for a workload will be.

The amount of queues is the amount of I/O that each thread is generating at once (OIO - Outstanding IO). Both of the default random I/O tests are set to 4K block size. While 4K is a block size that can be seen at times, it is usually not a common block size in most workloads today. When possible the block size in the test should match the application's block size. The common block size for both Oracle and MS SQL is 8 KB as well as 64 KB. While these applications have variable block sizes dependent on the work happening, 8 KB and 64 KB are very commonly used block sizes in the field.CrystalDiskMark tests settings can be adjusted in the "Settings" menu. CrystalDiskMark allows a user to specify the following parameters:


	Type (sequential or random)
	
	Block Size
	
	Queues
	
	Threads
	

 

Suggested example for testing Nutanix and other enterprise storageAlways create a new vDisk for the VM before running the tests. The main reason for that is the old disk might have some data from the previous tests or other workloads and it could have been down migrated to the HDD tier (in case of a hybrid cluster). The new disk will guarantee the correct data path of the I/O.Create a 50 GiB or 100 GiB disk. The disk then should be initialized and formatted with the default settings. The disk will get formatted and will get a letter assigned. Make sure that the new disk is selected in the CrystalDiskMark for the tests.NOTE: Nutanix scales performance per node and with the number of vDisks. The testing with CrystalDiskMark is based on a single vDisk on a single VM running on a single Node. While testing is important to understand a specific use case for a single vDisk, it should be clear that CrystalDiskMark does not replace a distributed performance test like Nutanix X-Ray to understand the full capability of a Nutanix cluster.The following settings are recommended as they provide a good mix of block sizes as well as parallel IO vs. single threaded IO:Figure 3: Recommended CrystalDiskMark settingsThe below table explains the different configuration details and differences to the default settings:


			Default tests:
			
			Recommended tests:
			
			Explanation:
			
			Sequential reads and writes of:

			
				Block size: 1 MB
				
				Queue depth: 8
				
				Threads: 1
				
			
			Sequential reads and writes of:

			
				Block size: 1 MB
				
				Queue depth: 8
				
				Threads: 8
				
			
			The test is very similar to the default sequential 1MB test, the only adjustment is the amount of threads has been changed from 1 to 8.			Such a test would be similar to what happens during a large file copy or during a backup job.
			
			Sequential reads and writes of:

			
				Block size: 1 MB
				
				Queue depth: 1
				
				Threads: 1
				
			
			Sequential reads and writes of:

			
				Block size: 64 KB
				
				Queue depth: 16
				
				Threads: 8
				
			
			As the default single-threaded 1MB test won't give much more insight with a queue depth of 1			Nutanix recommends replacing this test with a 64KB block size sequential test as it is more common for applications to use that block size.			The queue depth of 16 and 8 threads will ensure that the test generates enough load on the storage subsystem.
			
			Random reads and writes of:

			
				Block size: 4 KB
				
				Queue depth: 32
				
				Threads: 1
				
			
			Random reads and writes of:

			
				Block size: 8 KB
				
				Queue depth: 32
				
				Threads: 8
				
			
			The default 4K block size tests do not reflect the real-life usage of the application. However, 8K block size is very common and is used by most databases. 			The queue depth of 32 and 8 threads will ensure that the test generates enough load on the storage subsystem.
			
			Random reads and writes of:

			
				Block size: 4 KB
				
				Queue depth: 1
				
				Threads: 1
				
			
			Random reads and writes of:

			
				Block size: 8 KB
				
				Queue depth: 1
				
				Threads: 1
				
			
			The default 4K block size tests do not reflect the real-life usage of the application. However, 8K block size is very common and is used by most databases.			This test is left with a single thread and a queue depth of 1 just to show the difference between the multi-threaded test above and the single-threaded result.
			

Table1: Explanation of default vs. recommended CrystalDiskMark settingsA good Measure Time recommendation is 30 seconds. The default setting of 5 seconds won't generate enough load for that short time frame.The good Interval Time recommendation is 60 seconds. That is to make sure that there is a quiet time between write tests. Nutanix starts flushing the write cache after 30 seconds of inactivity, so a default interval of 5 seconds is not a good setting. The test wants to ensure that the write cache is empty before running the next test to achieve fair results. While this is Nutanix specific, an application would rarely read the same data after it has been written. These settings reflect a realistic ramp up time which can be similarly used on other Enterprise Storage Systems.A good recommendation for a testing file size: 16 GiBThat way the file is big enough to bypass read/write caches and test Storage performance rather than a cache.Example of test results with recommended settingsThe results will greatly vary depending on the underlying hardware configuration and software versions. The below results were achieved on the following setup:

VM details: Windows Server 2019 Datacenter Build 17763vCPUs: 8Memory: 8GbDisk size: 100Gb

Hardware details:3-node cluster NX-3061 (G3).2xSSDs 800Gb and 4 HDDs 1Tb per node

Nutanix AOS version: 5.19.1.5Hypervisor: AHV 20201105.12Storage container settings:Inline compression - onDeduplication - offErasure Coding (EC-x) - off

.Figure 4: CrystalDiskMark results. For throughput equivalents of the results, simply multiply the IOPS by the block size (i.e. 8k, 64k, 1M). For example, 1629 IOPS x 1M = 1629MB/sec.

Data randomness.

Data randomness is quite important because the storage engine of Nutanix will be compressing the file internally if compression is enabled on the storage container level. CrystalDiskMark has two options for laying the file:


	random (default)
	
	All 0x00 (0 Fill) 
	

This can be configured in Settings - Test Data. It is generally recommended to use the random pattern to get realistic results because Nutanix will optimize the NULL byte files and testing on NULL byte files will not reflect the performance of real applications.Figure 5: CrystalDiskMark data pattern

The difference can be demonstrated by running a test in each of the modes and then compressing the file into a .zip archive using the built-in Windows compression tool. Even though the compression algorithm is different on the Nutanix storage level, it will still show the difference. So, if the test file is 1Gb in size, we see a 1Gb file in Windows Explorer:Figure 6: The test file sizeIf the file is created with the default random pattern, compressing the file into .zip will result in the same file size:Figure 7: The test file compressed into .zip size with a random data patternHowever, if the file is created with the All 0x00 pattern and then compressed into .zip, the archive will be taking ~1Mb of storage space instead of 1Gb, because the 0s will be mostly eliminated by the compression algorithm:Figure 8: The test file compressed into .zip size with All 0x00 data patternA similar compression happens on the storage side and without true data randomness, the test file will utilise a lot less of the storage space and will likely fit more into the cache, which will result in an unfair test.Unlike the diskspd tool, CrystalDiskMark creates a file with a random data pattern by default. There are no tweaks needed to achieve randomness. 

File system software cachingUnlike the default settings of diskspd, CrystalDiksMark is disabling the software caching in its tests. This will show the same amount of IOPS/Throughput on the Nutanix system side as we see in the CrystalDiskMark test.Test example in CrystalDiskMark:Figure 9: CrystalDiskMark vs. Nutanix Storage PerformanceThe results from CrystalDiskMark vs. the Nutanix storage are very close. The reason for this difference is below example shows a single iteration at the moment of the screenshot capture and the CrystalDiskMark shows the average of the whole test.Reads:Figure 10: Nutanix Storage Performance read Writes:Figure 11: Nutanix Storage Performance write NOTE: CrystalDiskMark is an easy to use benchmarking tool which is generally a simplified version of diskspd with GUI. It lacks some control and settings compared to diskspd, so there is less tuning possible. If the settings are correctly set, the results will be realistic.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.