vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Why am I running out of SNAT ports for egress connections from pods in my cluster? 

 What’s happening 

As of IBM Cloud Kubernetes Service 1.25 the source Network Address Translation (NAT) port range has change to 32768 - 65535, where previously it was 1024 - 65535. This change was made to resolve possible issues where the following scenarios might occur.



* An SNAT port is chosen in a VPC cluster that conflicts with a NodePort of a LoadBalancer of type NLB. In this case, that egress connection fails.
* An SNAT port is chosen for a long running egress connection in the NodePort range of 30,000 - 32,767 that a cluster service later wants to use for a NodePort. In this case, that cluster service won't get traffic on that NodePort until the connection using that SNAT port is finished and has been closed.
* An SNAT port is chosen for a long running egress connection that a Linux service or hostPort pod port or hostNetwork pod port later wants to use. In that case, that service or pod won't start or work until the connection that is using that SNAT port is finished and has been closed.



 Why it’s happening 

Usually, limiting this port range is fine. However, if you are running a highly scalable application in pod-network pods, where on a single worker you have either of the following cases.



* Over 30,000 egress connections to destinations other than pods or nodes in the cluster open at once.
* 30,000 egress connections are being opened within a few minutes of each other.



 How to fix it 

It might be possible that the 32768 - 65535 range isn't large enough. Two possible solutions to this are the following options.



* Add more nodes and have at least one pod per node that make all these egress connections so that each node needs less than 30,000 SNAT ports. This is the preferred solution
* Explicitly set the pod natPortRange in Calico to a larger range.
Whether or not fuzzy matching is enabled, system entities always have a confidence level of 1.0. You can use this property in a condition to have it return false if the confidence level is not higher than a percent you specify. 
 location A zero-based character offsets that indicates where the detected entity values begin and end in the input text. Use .literal to extract the span of text between start and end index values that are stored in the location property. 
 value The entity value identified in the input. This property returns the entity value as defined in the training data, even if the match was made against one of its associated synonyms. You can use .values to capture multiple occurrences of an entity that might be present in user input. 







 Entity property usage examples 

In the following examples, the skill contains an airport entity that includes a value of JFK, and the synonym 'Kennedy Airport". The user input is I want to go to Kennedy Aiport.



* To return a specific response if the 'JFK' entity is recognized in the user input, you could add this expression to the response condition: entities.airport[0].value == 'JFK' or @airport = "JFK"
* To return the entity name as it was specified by the user in the dialog response, use the .literal property: So you want to go to <?entities.airport[0].literal?>... or So you want to go to @airport.literal ...

Both formats evaluate to So you want to go to Kennedy Airport... in the response.
* Expressions like @airport:(JFK) or @airport.contains('JFK') always refer to the value of the entity (JFK in this example).
* To be more restrictive about which terms are identified as airports in the input when fuzzy matching is enabled, you can specify this expression in a node condition, for example: @airport && @airport.confidence > 0.7. The node will only execute if your assistant is 70% confident that the input text contains an airport reference.



In this example, the user input is Are there places to exchange currency at JFK, Logan, and O'Hare?



* To capture multiple occurrences of an entity type in user input, use syntax like this:
Why am I running out of SNAT ports for egress connections from pods in my cluster? 

 What’s happening 

As of Red Hat OpenShift on IBM Cloud 4.12 the source Network Address Translation (NAT) port range has change to 32768 - 65535, where previously it was 1024 - 65535. This change was made to resolve possible issues where the following scenarios might occur.



* An SNAT port is chosen in a VPC cluster that conflicts with a NodePort of a LoadBalancer of type NLB. In this case, that egress connection fails.
* An SNAT port is chosen for a long running egress connection in the NodePort range of 30,000 - 32,767 that a cluster service later wants to use for a NodePort. In this case, that cluster service won't get traffic on that NodePort until the connection using that SNAT port is finished and has been closed.
* An SNAT port is chosen for a long running egress connection that a Linux service or hostPort pod port or hostNetwork pod port later wants to use. In that case, that service or pod won't start or work until the connection that is using that SNAT port is finished and has been closed.



 Why it’s happening 

Usually, limiting this port range is fine. However, if you are running a highly scalable application in pod-network pods, where on a single worker you have either of the following cases.



* Over 30,000 egress connections to destinations other than pods or nodes in the cluster open at once.
* 30,000 egress connections are being opened within a few minutes of each other.



 How to fix it 

It might be possible that the 32768 - 65535 range isn't large enough. Two possible solutions to this are the following options.



* Add more nodes and have at least one pod per node that make all these egress connections so that each node needs less than 30,000 SNAT ports. This is the preferred solution
* Explicitly set the pod natPortRange in Calico to a larger range.
Table 2. HTTP status codes

 Code Description 

 201 Database created successfully. 
 202 The database was successfully created on some nodes, but the number of nodes is less than the write quorum. 
 400 Invalid database name. 
 412 Database exists. 



See the following example response that is received after a database is created successfully:

HTTP/1.1 201 Created

{
"ok": true
}





 Database topology 

It's possible to modify the configuration of a sharding topology for a database on dedicated database clusters. This modification can be done when the database is created. However, poor choices for configuration parameters can adversely affect database performance.

For more information about modifying database configuration in a dedicated database environment, contact IBM Cloudant support.

It isn't possible to modify the configuration that is used for databases on multi-tenant clusters.
Review the file I/O bandwidth from the last day or week time window to determine the average bandwidth. You can use the average bandwidth as a threshold to set an alert for when the file I/O bandwidth is less than the average for a certain amount of time. For example, if the average file I/O bandwidth for your app is 300B/s, you can create an alert for when the network utilization remains less than 300B/s for a certain amount of time.
5. In the File I/O Bandwidth by Pod tile, create an alert by clicking the Options menu and then Create alert.
6. Open the Notification section of the alert menu and create or select an alert notification channel.
7. Save your alert.
8. Repeat these steps for all the apps that are deployed in your cluster.
9. Test the alert that you created by editing the threshold that you configured to manually trigger the alert. For example, if you set a file system utilization alert to trigger when the utilization is less than 300B/s for 5 minutes, increase the threshold to be greater than the current utilization of your app for 5 minutes and select the at least once option.
10. Verify that the alert is triggered after 5 minutes. Once you have verified the alert, reset the values to the values you configured earlier.



5. Create a network utilization alert for an app that runs in your cluster.



1. From the IBM Cloud Monitoring console, click Overview > Workloads.
2. Select the Namespace where your app is deployed. Find your app, click the arrow icon on your app, and select Kubernetes Pod overview.
3. In the Network Utilization section, review the Network Request Count by Pod tile.
4. Review the average network request count by pod from the last 1 day or 1 week time window to determine the threshold.
5. In the Network Request Count by Pod tile, create an alert by clicking the Options menu and then Create alert. Set the alert parameters based on the threshold that you observed. For example, if the network utilization remains under the threshold for a certain amount of time, then the alert is triggered.
6.
15 is the maximum number of subnets that you can define with your ALB.
* Is the load balancer horizontally scalable?

Yes. The Application Load Balancer for VPC automatically adjusts its capacity based on the load. When horizontal scaling takes place, the number of IP addresses associated with the application load balancer's DNS changes.
* Why is my application load balancer in maintenance_pending state?

The application load balancer is in maintenance_pending state during various maintenance activities, such as:



* Horizontal scaling activities
* Recovery activities
* Rolling upgrades to address vulnerabilities and apply security patches



* Why do I need to choose multiple subnets during provisioning?

The Application Load Balancer for VPC (ALB) is Multi-Zone Region (MZR) ready. Load balancer appliances are deployed to the subnets you selected. To achieve higher availability and redundancy, deploy the application load balancer to subnets in different zones.
* Do I need extra IPs in the subnet for application load balancer operations?

It is recommended to allocate 8 extra IPs per MZR to accommodate horizontal scaling and maintenance operations. If you provision your application load balancer with one subnet, allocate 16 extra IPs.
* What are the default settings and allowed values for health check parameters?



* Health check interval - Default is 5 seconds, and the range is 2 - 60 seconds.
* Health check response timeout - Default is 2 seconds, and the range is 1 - 59 seconds.
* Maximum retry attempts - Default is two retry attempts, and the range is 1-10 retries.



The health check response timeout value must be less than the health check interval value.
* Are the ALB IP addresses fixed?

Application load balancer IP addresses are not guaranteed to be fixed. During system maintenance or horizontal scaling, you see changes in the available IPs associated with the FQDN of your load balancer.

Use FQDN, rather than cached IP addresses.
* Does the load balancer support layer 7 switching?

Yes, the load balancer supports layer 7 switching.
* Why does HTTPS listener creation or update tell me that my certificate is invalid?
[Screen capture of test dialog output](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/test_dialog_output.png)
4. To see the top intents that were recognized in the test input along with their confidence scores, hover over the eye icon that is displayed next to the intent with the highest confidence score.

![Screen capture of the eye icon that is shown next to the top-most recognizes intent](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/tryit-intent-details.png)
5. If the response is not what you expected it to be, you can take the following actions from the Try it out pane:



* If you want to edit an entity that is recognized in the input, click the entity name to open it in the Entities page.
* If the wrong intent is recognized, you can click the arrow next to the intent name to correct it or mark the topic as irrelevant. For more information, see [Making training data improvements](https://cloud.ibm.com/docs/assistant?topic=assistant-logslogs-fix-data).



6. If you want to know which node in the dialog tree triggered a response, click the Location![Location](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/location.png) icon next to it.

If you are not already on the Dialog page, open it.
7. To check or set the value of a context variable, click the Manage context link.

Any context variables that you defined in the dialog are displayed.

In addition, a $timezone context variable is listed.
To take advantage of the auto-scale feature from [Host Factory](https://www.ibm.com/docs/en/spectrum-symphony/7.3.1?topic=factory-overview), the value needs to be greater than the worker_node_min_count. Enter a value in the range 1 - 500. if you are using bare metal servers, the value needs to match the worker_node_min_count, and the permitted value is in the range 1 - 16. Note: If you plan to use Spectrum Scale storage, the value for this parameter should be equal to worker_node_min_count. No 10 
 worker_node_min_count The minimum number of virtual server instance or bare metal server worker nodes that will be provisioned at the time the cluster is created. For bare metal servers, enter a value in the range 1 - 16. For virtual server instances with NFS storage, enter a value in the range 0 - 500. For virtual server instances with Spectrum Scale storage, enter a value in the range 1 - 64. Note: Spectrum Scale requires a minimum of 3 compute nodes (combination of primary, secondary, management, and worker nodes) to establish a [quorum](https://www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=failure-quorumnodequo) and maintain data consistency if a node fails. Therefore, the minimum value of 1 might need to be larger if the value specified for management_node_count is less than 2. No 0 
 worker_node_type The type of server that's used for the worker nodes: virtual server instance or bare metal server. If you choose vsi, the worker nodes are deployed on virtual server instances, or if you choose baremetal, the worker nodes are deployed on bare metal servers. Note: If baremetal is selected, only static worker nodes are supported; you will not be able to use the Spectrum Symphony Host Factory feature for auto-scaling on the cluster. No vsi 
 zone IBM Cloud zone name within the selected region where the IBM Spectrum Symphony cluster should be deployed.
* Worker nodes ran out of CPU, memory, or disk space.
* Maximum pod limit per cluster reached.
* App itself has an issue.



To set up monitoring for these conditions, configure alerts based on the following IBM Cloud Monitoring metrics. Note that your alert thresholds might change depending on your cluster configuration.



App level metrics

 Metric IBM Cloud Monitoring metric Alert threshold 

 Multiple restarts of a pod in a short amount of time. kubernetes_pod_restart_count Greater than 4 for the last 10 minutes 
 No running replicas in a ReplicaSet. kube_replicaset_status_replicas in kubernetes_deployment_name Less than one. 
 More than 5 pods pending in cluster. kube_pod_container_status_waiting Status equals pending greater than five. 
 No replicas in a deployment available. kubernetes_deployment_replicas_available Less than one. 
 Number of pods per node reaching threshold of 110. Count by (kube_cluster_name,kube_node_name)(kube_pod_container_info) Greater than or equal to 100. Note that this query is a promQL query. 
 Workloads that are in an unknown state. (kube_workload_status_unavailable) Greater than or equal to one. Note that this query is a promQL query. 







 Worker node alerts 

Review the following thresholds and alerts for worker nodes.



Worker node metrics

 Metric IBM Cloud Monitoring metric Alert threshold 

 CPU utilization of the worker node over threshold. cpu_used_percent Greater than 80% for 1 hour. 
 CPU utilization of the worker node over threshold. cpu_used_percent Greater than 65% for 24 hours. 
 Memory utilization of the worker node over threshold. memory_used_percent Greater than 80 % for 1 hour. 
 Memory utilization of the worker node over threshold. memory_used_percent Greater than 65% for 24 hours. 
 Amount of memory used over threshold. memory_bytes_used Greater than NUMBER_OF_BYBTES. 
 Nodes with disk pressure exist.
/dev/mapper/XXXp1 /PerfDisk ext3 defaults,_netdev 0 1











 Verifying MPIO configuration 

If MPIO isn't configured correctly, your storage device might disconnect and appear offline when a network outage occurs or when IBM Cloud® teams perform maintenance. MPIO ensures an extra level of connectivity during those events, and keeps an established session to the LUN with active read/write operations.



* To check whether multipath is picking up the devices, list the current configuration. If it is configured correctly, then each volume has a single group, with a number of paths equal to the number of iSCSI sessions.

multipath -l

root@server: multipath -l
3600a09803830304f3124457a45757067 dm-1 NETAPP,LUN C-Mode
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| 6:0:0:101 sdd 8:48 active ready running
-+- policy='round-robin 0' prio=-1 status=enabled
- 7:0:0:101 sde 8:64 active ready running

The string 3600a09803830304f3124457a45757067 in the example is the unique WWID of the LUN. Each volume is identified by its unique WWID, which is persistent while the volume exists.
* Confirm that all the disks are present. In a correct configuration, you can expect two disks to show in the output with the same identifier, and a /dev/mapper listing of the same size with the same identifier. The /dev/mapper device is the one that multipath sets up.

fdisk -l | grep Disk



* The following example output shows a correct configuration.



root@server: fdisk -l | grep Disk
During creation of the IBM Cloud Kubernetes Service cluster, it is expected that you completed the steps to also connect to a Log Analysis service.



1. From the [Kubernetes clusters](https://cloud.ibm.com/kubernetes/clusters), click on the name of the Kubernetes cluster you just created and click Overview on the left pane.
2. Scroll down to Integrations and open the Log Analysis UI by clicking Launch. It may take a few minutes before you start seeing logs.

If instead of Launch you see a Connect button, you can click on it to create the integration if it was not done during the creation of the cluster. It simplifies the installation of logdna-agent pod on each node of your cluster. The logging agent reads log files from the pod where it is installed, and forwards the log data to your logging instance.
3. To check whether the logdna-agent pods on each node of your cluster are in Running status, run the below command in a shell:

kubectl get pods --namespace ibm-observe

You should see an output similar to the one below, with one logdna-agent running per worker node that you have deployed in your cluster.

NAME READY STATUS RESTARTS AGE
logdna-agent-4nlsw 1/1 Running 0 39s
logdna-agent-lgq9f 1/1 Running 0 39s
logdna-agent-ls6dc 1/1 Running 0 39s







 Step 4: Generate and access application logs 

In this section, you will generate application logs and review them in Log Analysis.



 Generate application logs 

The application deployed in the previous steps allows you to log a message at a chosen log level. The available log levels are critical, error, warn, info and debug. The application's logging infrastructure is configured to allow only log entries on or above a set level to pass. Initially, the logger level is set to warn. Thus, a message logged at info with a server setting of warn would not show up in the diagnostic output.
4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored 1 Pass IBM 
 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive Scored 1 Pass IBM 
 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root Scored 1 Pass IBM 
 4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive Scored 1 Pass IBM 
 4.1.10 Ensure that the kubelet configuration file ownership is set to root:root Scored 1 Pass IBM 







 4.2 Kubelet 



Section 4.2 Kubelet benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 4.2.1 Ensure that the --anonymous-auth argument is set to false Scored 1 Pass IBM 
 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored 1 Pass IBM 
 4.2.3 Ensure that the --client-ca-file argument is set as appropriate Scored 1 Pass IBM 
 4.2.4 Ensure that the --read-only-port argument is set to 0 Scored 1 Pass IBM 
 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 Scored 1 Pass IBM 
 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) IBM 
 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true Scored 1 Pass IBM 
 4.2.8 Ensure that the --hostname-override argument is not set Not Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) IBM 
 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored 2 Pass IBM
4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored 1 Pass IBM 
 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive Scored 1 Pass IBM 
 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root Scored 1 Pass IBM 
 4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive Scored 1 Pass IBM 
 4.1.10 Ensure that the kubelet configuration file ownership is set to root:root Scored 1 Pass IBM 







 4.2 Kubelet 



Section 4.2 Kubelet benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 4.2.1 Ensure that the --anonymous-auth argument is set to false Scored 1 Pass IBM 
 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored 1 Pass IBM 
 4.2.3 Ensure that the --client-ca-file argument is set as appropriate Scored 1 Pass IBM 
 4.2.4 Ensure that the --read-only-port argument is set to 0 Scored 1 Pass IBM 
 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 Scored 1 Pass IBM 
 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true Scored 1 Pass IBM 
 4.2.8 Ensure that the --hostname-override argument is not set Not Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored 2 Pass IBM
The default value is 0.
* Maximum number of active crawler threads

The default value is 10.
* Maximum number of documents to crawl

The default value 2000000000.
* Maximum document size (KB)

The default value is 32768.



Data source properties
: Complete the following fields



* Host name

The default value is localhost.
* Port

The default value is 22.
* User name
* Use key file (or input password)

The default value is On.
* Key file location
* passphrase
* Password



Crawl Space Properties
: If the custom crawler supports document-level security and the document_level_security_supported value in the template.xml is set to true, then an Enable Document Level Security switch is displayed in a Security section of the data source connection setup page. To enable document-level security, set the Enable Document Level Security switch to On. If the switch is set to Off, then the collection that is created cannot support document-level security even if the custom crawler can support document-level security.
4. Click Finish to create the collection.