rghosh8/supportGPT_data huggingface dataset:
### Human: Can you tell about Configuring AOS for VMware Fault-Tolerance on ESXi 6.7. ### Assistant: AOS 5.19.x now supports VMware Fault Tolerance running ESXi 6.7. The Nutanix cluster requires configuring two Stargate timeouts as well as RPC timeouts on ESXi to ensure the timely failover of NFS services.. Starting from AOS 5.19.x, VMware Fault Tolerance is now supported with ESXi 6.7. To enable this feature, two Stargate timeouts, as well as RPC timeouts, must be configured on ESXi hosts to ensure timely failover of NFS services as required to support Fault Tolerant VMs.

In addition to the two Stargate timeouts, the following two parameters value must be increased to 30 seconds on each ESXi host in a cluster:

/NFS/CreateRPCTimeout/NFS/RenameRPCTimeout. Contact Nutanix Support for assistance to set the Stargate timeouts. This configuration change requires a rolling restart of the Stargate process on each node of the cluster.

To set the ESXi host parameters /NFS/CreateRPCTimeout and /NFS/RenameRPCTimeout follow instructions below:

To set the parameters:

[root@ESXi]# esxcfg-advcfg -s 30 /NFS/CreateRPCTimeout

To check the parameters:

[root@ESXi]# esxcfg-advcfg -s 30 /NFS/CreateRPCTimeout. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Nutanix user being locked out from scale-out PC. ### Assistant: Nutanix user is locked on scale out Prism Central. Users cannot login as "nutanix" via SSH to a scale-out Prism Central deployment.
As an admin user via SSH, running the below command, it's possible to notice that the "nutanix" user is locked due to several "su -l" commands:

admin@PCVM:~$ sudo faillock --user nutanix| tail -15

Switch to root with "sudo su -" and look into /home/log/secure logs for the below error message:

admin@PCVM:~$ sudo faillock --user nutanix| tail -15

Another way to identify the issue is to check /home/log/messages file:

admin@PCVM:~$ sudo faillock --user nutanix| tail -15

By tracing one of the failed PIDs in sysstats top.INFO log, it points to the below command:

admin@PCVM:~$ sudo faillock --user nutanix| tail -15

Which turns out to be a command run every minute by the vip_cleanup script:

admin@PCVM:~$ sudo faillock --user nutanix| tail -15
** NOTE **: The script only blocks 2 PCVMs out of 3 since it does not run on the PCVM with the VIP. This is also why it only affects some PCVMs, as there's always a VIP in scale-out deployments. If needed, it's possible to unlock the "nutanix" user with the below command in the affected PCVM: 

admin@PCVM:~$ sudo faillock --user nutanix| tail -15. This issue is resolved in pc.2022.6.0.6. Nutanix recommends that the user upgrade Prism Central to pc.2022.6.0.6 or higher. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Prism Central slowness as it receives a huge amount of incoming v1/users/details API calls from PE Clusters causing availability issues. ### Assistant: Prism Central deployment with a large count (hundreds) of managed PE nodes (CVMs) may experience prism gateway service availability problems due to the constant flow of incoming PrismGateway/services/rest/v1/users/details API calls coming from Prism Elements with very high frequency.. Prism Central deployment with a large count (hundreds) of managed PE nodes (CVMs) may experience prism gateway service availability problems due to the constant flow of incoming PrismGateway/services/rest/v1/users/details API calls coming from Prism Elements with very high frequency.Identification:Verify that Prism Central manages multiple Prism Elements with a large total count of nodes in connected PEs (hundreds); To check the total number of managed PE nodes, use the command below:
nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
Symptoms may include:

The constant incoming flow of v1/users/details API calls in prism_proxy_access_log.out from connected Prism Elements. The frequency of incoming v1/users/details calls is very high (hundreds per minute)
	nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
	Prism Gateway service may be killed with OOM:
	nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
	You may see frequent entries of pods getting killed with OOM:
	nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
	Some API calls to Prism Gateway service may return HTTP response code 429 Too Many Requests:
	nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
	IAM enabled Prism Central may experience iam-proxy pod OOM restarts:
	nutanix@PCVM:~$ /home/docker/msp_controller/bootstrap/msp_tools/cmsp-scripts/idfcli get entity -e node -a | grep entity_id | wc -l
	
On every Prism Eelement, CVM's ikat_control_plane service sends an excessive number of PC availability checks that cause a DDOS-like effect for PCs managing hundreds of nodes resulting in this issue. Note: Not always the above symptoms are caused by the problem discussed in this article. Technically any source of API requests incoming to PCVM with high frequency may cause similar issues. This KB article is relevant only for PC deployments managing hundreds of nodes and experiencing problems due to connected PEs sending large counts of v1/users/details API calls.. This issue is resolved in AOS versions 6.6.1, 6.5.2, and above. Upgrade the connected Prism Element clusters to the fix versions.
 . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about How to create long living tokens to integrate CI/CD pipeline in Nutanix Kubernetes Engine. ### Assistant: How to create long living tokens to integrate CI/CD pipeline.. Nutanix Kubernetes Engine is formerly known as Karbon or Karbon Platform Services. Nutanix Kubernetes Engine default token is valid for only 24 hours, which makes integration difficult with external components like CI/CD pipeline and Kubernetes cluster deployed by Nutanix Kubernetes Engine. . Nutanix recommend using the Nutanix Kubernetes Engine API to regenerate the kubeconfig before the 24 hours expiry and integrate that process in your CI/CD workflow. See Nutanix Dev for reference.Alternatively, you can use Nutanix Kubernetes Engine Service Account. This is not recommended as it requires an experienced administrator.In the procedure below, we are going to create a service account for Jenkins integration.

For the sake of simplicity, admin privilege is assigned via ClusterRole.More restricted access can be assigned using RBAC.

Create a service account
	$ kubectl create serviceaccount jenkins
	Create a role binding based on the permission needed by the application
	$ kubectl create serviceaccount jenkins
	Extract Service account token.
	$ kubectl create serviceaccount jenkins
		Clusters running Kubernetes > 1.24:	Starting Kubernetes 1.24, Kubernetes won't generate Secrets automatically any longer for ServiceAccounts. This means the command from step 3 will return an empty output on k8s > 1.24	If the cluster is running  Kubernetes 1.24 or above, use the alternative step 3a:		3a. Create a service account token secret or create the token manually for the ServiceAccount	Option 1: Create a secret of type service-account-token
	$ kubectl create serviceaccount jenkins
		Then extract the token
	$ kubectl create serviceaccount jenkins
		Option 2: Create a token manually for the Service Account:
	$ kubectl create serviceaccount jenkins
	Download a new kubeconfig file from Karbon.Update the token in the kubeconfig file with the token we generated in the steps above.Use the modified kubeconfig in Jenkins pipe to integrate the k8s cluster.

$ kubectl create serviceaccount jenkins
Important Notes:

Nutanix recommends using kubeconfig from Rest API or UI for User logins.If you have to use Service accounts limit restrict the use to service integration like CI/CD pipeline. Avoid using service accounts for general-purpose tasks.Distributing long-lived tokens to users may introduce secret sprawl, thus, administrators must ensure that the tokens are not used for unintended purposes.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Adding or Removing a PCI device causes LSI controller to disconnect.. ### Assistant: Currently if you add a PCI device, physical or virtual, to an ESXi host the PCI bus number enumeration will change. This causes the LSI Adapter to go into a disconnected state on the Nutanix CVM.. Currently if you add a PCI device, physical or virtual to an ESXi host the PCI bus number enumeration will change. This causes the LSI Adapter to go into a disconnected state and the CVM will not power on.

UCS is inherently susceptible to this as you can add and remove as many vNIC's and vHBA's to a service profile as you need, but this has also been observed on the NX-8150 when adding a 10GB NIC card. It may also occur if a customer adds a new 10GB NIC if they initially only purchased and set up the cluster with 1GB NIC's.

To identify this, check on the individual hosts configuration under Configuration > Advanced Settings you will see the message "No devices currently enabled for passthrough"



The CVM itself will not power on. If you look at the VM settings you will see "Unavailable Device" for the PCI attached device 0.. To resolve this, in the respective host vSphere instance, go to the Host Configuration > Advanced Settings page and click Configure Passthrough on the right-hand side. From here, find the LSI Logic (LSIModel) device(s) / ID(s), check the box and click Ok to enable it. Some nodes, like the NX-8150-G5, can have one to three LSI HBA cards.

Next, reboot the host for the configuration to take effect. After reboot, the controller should be listed as below.Once the host is back up, edit the CVM configuration in vCenter. Remove the old PCI device(s), and add the new one(s). Select the LSI adapter(s) where PCI Passthrough just made active, and save.After hitting OK, power on the CVM.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Prism Central - pc.2021.9.0.5, pc.2022.1, or later: Unable to access PE via Cluster Quick Access after upgrading the AOS of the PE. ### Assistant: After upgrading AOS in a Prism Element which has been managed by a Prism Central that version is pc.2021.9.0.5, pc.2022.1 or later, you may not be able to access the Prism Element by using Cluster Quick Access feature of the Prims Central.. After upgrading an AOS cluster that is managed by a Prism Central running version pc.2021.9.0.5, pc.2022.1, or later, the user may not be able to access the Prism Element by using the Cluster Quick Access feature of the Prism Central.Behind the scene, the Prism service in Prism Central downloads the new Prism UI data from the upgraded Prism Element. This Prism service may have downloaded the new Prism UI data from PE but may not extract it for Cluster Quick Access use.
In the /home/nutanix/data/logs/prism_gateway.log in the Prism Central VM, you may find the ProtobufZNodeManagementException$BadVersionException exception happening with the following stack trace almost every minute.

nutanix@PCVM:-$ cat ~/data/logs/prism_gateway.log | grep 'ProtobufZNodeManagementException$BadVersionException'

If the Prism service has been restarted, you may see this ProtobufZNodeManagementException$BadVersionException once after the restart, and also you may see NullPointerException happening almost every minute, as the followings.

nutanix@PCVM:-$ cat ~/data/logs/prism_gateway.log | grep 'ProtobufZNodeManagementException$BadVersionException'
 

nutanix@PCVM:-$ cat ~/data/logs/prism_gateway.log | grep 'ProtobufZNodeManagementException$BadVersionException'

Users may also notice that the Prism service downloaded the Prism UI data from the upgraded PE, but no "console" directory is created.For example, if you have upgraded your PE to AOS 5.20.2, Prism Central will download the "PrismUI.tar.gz" file from the PE, and then this file will be put in the "/home/apache/www/console/el7.3-release-euphrates-5.20.2-stable-9fe17e864a79fb1f4e58eda7a12075f9427ca364" directory. Then Prism Central will extract this file into the "console" directory in the same directory as the "PrismUI.tar.gz" file. But, this "console" directory is not created in this case.

nutanix@PCVM:-$ cat ~/data/logs/prism_gateway.log | grep 'ProtobufZNodeManagementException$BadVersionException'

You may find the delete_old_prismui_files.py script had run and deleted old unnecessary Prism UI data files before the AOS upgrade in the "/home/nutanix/data/logs/delete_old_prismui_files.log" file.

nutanix@PCVM:-$ cat ~/data/logs/prism_gateway.log | grep 'ProtobufZNodeManagementException$BadVersionException'. This issue has been fixed in these Prism Central releases.
pc.2021.9.0.6pc.2022.6.0.6, or later
Please upgrade Prism Central before carrying out an AOS upgrade on PE clusters. If you have already been experiencing this issue, please get in touch with Nutanix Support for recovery.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about NCC Health Check: vmknics_subnet_check. ### Assistant: The NCC health check vmknics_subnet_check verifies if vmknics on ESXi host have IP addresses in the same IP subnet.. The NCC vmknics_subnet_check test verifies if vmknics IP addresses are configured on the same subnet on any given host.

Note: vmknics in the same subnet on the same ESXi host is an unsupported configuration.

Running the NCC Check

Run this check as a part of the complete NCC checks.

nutanix@cvm$ ncc health_checks run_all

Or run the vmknics_subnet_check check individually.

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is not scheduled to run on an interval.

This check does not generate an alert.

Sample output

If multiple vmknics are found with IP addresses in the same subnet, the check reports a warning, and an output similar to the following is displayed.

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID106442DescriptionCheck if vmknics have different subnetsCauses of failurevmknics have ip address configured in the same subnetResolutionsCorrect the IP addressing in the network.			Review KB 2722 for more details.Impactvmknics in the same ip subnet on the same esxi host is unsupported.. If vmknics_subnet_check reports a WARN status, correct the IP address on the ESXi host that is reported by the NCC.

Note: Nutanix does not recommend using two vmknics in the same subnet on ESXi.

When the two vmknics reside on the same subnet, ESXi is unable to split vMotion traffic and other management traffic apart according to the GUI configuration. vMotion is a burst-type workload that uses no bandwidth until DRS or a vSphere administrator starts a vMotion (or puts a host into maintenance mode). But when a vMotion starts, the network interface gets saturated as the vMotion is also used for the Nutanix Cluster. Nutanix recommends using two different subnets.

Verify the vmknics configuration and rectify the configuration so that the vmknics belong to different subnets.

After the problem is resolved and the vmknics are in different subnets or one of the vmknics is removed, restart the CVM (Controller VM) to read the network configuration changes.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Nutanix Files: Unable to access TLDs after rename. ### Assistant: Nutanix Files 4.3 has a regression causing TLDs to be unaccessible after renaming.. Nutanix Files 4.3 has a regression that causes TLDs to be inaccessible after renaming them.  This only applies to TLDs created on Nutanix Files versions pre-3.5.Below is the error the client will get when accessing the TLD.

\\<File Server Name>\<Share Name>\<TLD Name> is unavailable. 
Verification:Run the below on any FSVM to verify that the FSVMs cannot locate the metadata for the TLD.  Replace <Name of TLD> with the actual name of the TLD
\\<File Server Name>\<Share Name>\<TLD Name> is unavailable. . This issue has been resolved in Nutanix Files 4.3.0.1.  Please upgrade to that version or later. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Nutanix Files - Recall request has failed. ### Assistant: This article describes an issue where recall request fails on a Nutanix Files cluster with tiering enabled after being upgraded to version 4.3. This results in files not being accessed from the s3 store.. The issue occurs on clusters with Tiering enabled after upgrading the Files cluster from ≥ 4.0 and < 4.3 to version 4.3. Recall request fails, resulting in files not being accessible from the s3 store.

Identifying the problem

In the "Data Lens Dashboard", under the Tasks, you see a failed "Recall Request".When you manually recall tiered data, the task either fails after a prolonged time or takes longer than usual to succeed.If the recall request is successful, the recalled file is partially full or cannot be opened due to an error.The base URL value of the tiering profile is prefixed with the bucket name as "bucket_name.endpoint":
	nutanix@FSVM$ afs tiering.list_profiles
	. Nutanix is aware of the issue and is working on a resolution. Contact Nutanix Support for immediate assistance.

Collecting additional information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB-2871.Run a complete NCC health_check on the cluster. See KB-2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*NVM logs stored within the NVM at ~/data/logs/minerva*To collect the file server logs, run the following command from the CVM. Ideally, run this on the Minerva leader as there were issues seen otherwise. To get the Minerva leader IP on the AOS cluster, run:
		nutanix@CVM$ afs info.get_leader

		Once you are on the Minerva leader CVM, run the following:

		nutanix@CVM$ afs info.get_leader

		For example:

		nutanix@CVM$ afs info.get_leader
		
	

Attaching files to the caseTo attach files to the case, follow KB-1294. If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about Deploying Witness VM on an ESXi cluster version 6.5 or earlier fails with the error "Selected OVA file is not a valid template". ### Assistant: This article describes investigates an issue with deploying a Witness VM on ESXi 6.5 and earlier.. Deploying Witness VM on an ESXi cluster version 6.5 or earlier will fail with the below message: 
Selected OVA file is not a valid template.
The issue is hardware incompatibility, as the Witness VM ESXi deployment OVA file is built from the ESXi 6.7 image.The issue applies only to new Witness VM deployment on ESXi 6.5 or earlier using the OVA file and does not apply to upgrading the Witness VM. The below Witness VM releases are not compatible with deployment on ESXi 6.5 and earlier.5.20, 5.20.1, 5.15.7, 5.15.6, 6.0, 6.0.1, 5.19.2 . Witness VM is forward and backwards compatible with AOS STS and LTS releases.Deploy the 5.15.5 Witness VM build located under the "Other Versions" tab on the Witness VM downloads page. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.
### Human: Can you tell about NCC Health Check: inode_usage_check. ### Assistant: The NCC check inode_usage_check verifies whether the number of free inodes on CVMs is getting low.. The NCC check inode_usage_check verifies whether the number of free inodes on CVMs is getting low.

An inode contains information (metadata) about a file in a file system, including where the data is stored, file name, file permissions and so on. Each directory and file has a corresponding inode, in case of a big file size – more than one inode. A maximum number of inodes for each filesystem is defined at FS creation time and cannot be increased.

The inode_usage_check returns the following statuses:

PASS if inode usage is at 75 percent or belowWARN if inode usage is between 75 and 90 percentFAIL if inode usage is above 90 percent

Running the NCC Check

It can be run as part of the complete NCC check by running:

nutanix@cvm$ ncc health_checks run_all

or individually as:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every 5 minutes, by default.This check will generate an alert A1027 after 1 failure across scheduled intervals.

Sample output

For Status: PASS

nutanix@cvm$ ncc health_checks run_all

For Status: WARN

nutanix@cvm$ ncc health_checks run_all

For Status: FAIL

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID1004DescriptionChecks if current inode usage is high.Causes of failureInode usage is high.ResolutionsReduce disk usage or replace disk.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert IDA1027Alert Smart TitleDisk Inode Usage High on Controller VM svm_ip_addressAlert TitleDisk Inode Usage HighAlert MessageInode usage for one or more disks on Controller VM svm_ip_address has exceeded inode_usage_threshold%.

 . The NCC health check inode_usage_check fails when one or more filesystems on disks is running out of free inodes or the overall cluster storage is running out of free inodes.

Ensure that the inodes are free on the CVM (Controller VM) which is reported in the failure section of the NCC check:

nutanix@cvm:~$ df -i

Example output (note the IUse% column):

nutanix@cvm:~$ df -i

If you observe that /home is indicating a high inode usage percentage, then determine which directory or directories have high inode usage using the following command:

nutanix@cvm:~$ df -i

Note: If you see any other directory (like /home/nutanix/data/stargate-storage/disks/<serial>) with high inode usage percentage, use the following command:

nutanix@cvm:~$ df -i

Once the above outputs are collected, engage Nutanix Support.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.