vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Preparing for host maintenance updates 

IBM engineers perform host maintenance to improve stability, provide security enhancements, and support upcoming new features. At times, IBM Cloud infrastructure providers perform maintenance on the hosts that house the Virtual Servers that are used as workers in your cluster, which may cause some of your workers to briefly go offline. However, there are actions you can take before the maintenance period that can minimize disruptions to your worker nodes. A notification with maintenance details and a list of affected workers is sent to customers before the maintenance window. Follow these steps to prepare your workers for an upcoming maintenance period.



 Identifying your affected workers 

If your workers are scheduled to undergo maintenance, you receive a notification before the maintenance window begins. A list of the workers that are affected is included in the notification.

The list of impacted components may look similar to the following example. The steps documented here apply to the workers listed in the IBM Kubernetes Service or Red Hat OpenShift on IBM Cloud Workers section.

Virtual Server Instances scheduled for maintenance

Virtual Server Instances in your account:

ID Name
1111_1a111aaa-1a1a-1aa1-1111-a11a111a1111 my_vpc_cluster_1

Virtual Server Instances in service accounts:

Application Load Balancer
alb-11aa111a-1111111

IBM Kubernetes Service or Red Hat OpenShift on IBM Cloud workers
kube-aaaa1aaa111aa11aa11a-aaaaaaaaaaa-aaaaaaa-00001a11
kube-aaaa2aaa222aa22aa22a-aaaaaaaaaaa-aaaaaaa-00002a22
kube-aaaa3aaa333aa33aa33a-aaaaaaaaaaa-aaaaaaa-00003a33





 Actions to take before the maintenance period 

Follow these steps to prepare your workers for the maintenance period. Workers can't be scheduled on hosts that are scheduled for maintenance. You can avoid disruptions to your workload by rebooting or replacing your workers so that they move to different hosts that are not undergoing maintenance.



 Workers in Classic clusters
Dev-VPC-Global VPC Base Create the base VPC for the development environment 
 Dev-Kub-Dallas Kubernetes Cluster Create the cluster in Dallas for development 
 Dev-Kub-London Kubernetes Cluster Create the cluster in London for development 
 Prod-VPC-Global VPC Base Create the base VPC for the production environment 
 Prod-Kub-Dallas Kubernetes Cluster Create the cluster in Dallas for production 
 Prod-Kub-London Kubernetes Cluster Create the cluster in London for production 
 Prod-Kub-Tokyo Kubernetes Cluster Create the cluster in Tokyo for production 
 Prod-Kub-Sydney Kubernetes Cluster Create the cluster in Sydney for production 



You can duplicate the development configuration in the project.json file and modify it as needed to quickly set up a second environment for production.





 Creating access groups for projects 

Access to projects is controlled by Identity and Access Management (IAM). It's recommended to create two or three access groups per project and assign users that work on the project to one of those access groups. For example, you might create a ProjectName-Reader access group that provides read-only access to the project for users who need to monitor costs or availability and a ProjectName-Writer access group for operations users who need to make changes to a project and deploy resources. If needed, you can create two writer access groups, one for users who can add configs and complete input values and a second for users who can deploy resources.



Table 2. Project access groups and roles

 Access group Roles 

 ProjectName-Reader Reader, Viewer 
 ProjectName-Writer Manager, Operator 



To create new projects, users must be assigned specific access on the project tooling service and IBM Cloud® Schematics service. For more information, see [Assigning users access to projects](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-access-project).





 Monitoring needs attention items 

Needs attention items are best used to monitor validation, approvals, failures, and version updates. By checking the needs attention items regularly, you can ensure that your project and configuration are up-to-date and compliant.





 Adding tags to projects
Why does my PVC remain in a pending state? 

[

This troubleshooting topic applies only to Red Hat OpenShift clusters that run version 3.11.

 What’s happening 

When you create a PVC and you run oc get pvc <pvc_name>, your PVC remains in a Pending state, even after waiting for some time.

 Why it’s happening 

During the PVC creation and binding, many different tasks are executed by the IBM Cloud Object Storage plug-in. Each task can fail and cause a different error message.

 How to fix it 

Describe your PVC and review the common error messages.



1. Find the root cause for why the PVC remains in a Pending state.

oc describe pvc <pvc_name> -n <namespace>
2. Review common error message descriptions and resolutions.





Block Storage error messages

 Error message Description Steps to resolve 

 can't get credentials: can't get secret <secret_name>: secrets "<secret_name>" not found The Kubernetes secret that holds your IBM Cloud Object Storage service credentials does not exist in the same namespace as the PVC or pod. See [PVC or pod creation fails due to not finding the Kubernetes secret](https://cloud.ibm.com/docs/openshift?topic=openshift-cos_secret_access_fails). 
 can't get credentials: Wrong Secret Type.Provided secret of type XXXX.Expected type ibm/ibmc-s3fs The Kubernetes secret that you created for your IBM Cloud Object Storage service instance does not include the type: ibm/ibmc-s3fs. Edit the Kubernetes secret that holds your IBM Cloud Object Storage credentials to add or change the type to ibm/ibmc-s3fs. 
 Bad value for ibm.io/object-store-endpoint XXXX: scheme is missing. Must be of the form http://<hostname> or https://<hostname>Bad value for ibm.io/iam-endpoint XXXX: scheme is missing.
curl https://sgmanager.<region>.securegateway.cloud.ibm.com
* If there is error when accessing tunnel server:

[DEBUG] The Secure Gateway tunnel is connecting for wss://<node>.securegateway.appdomain.cloud:9000/ws
[ERROR] The following error occurred on the Secure Gateway tunnel, <Error Code>
[INFO] The Secure Gateway tunnel was disconnected



* Please check whether your Secure Gateway Client have access to <node>.securegateway.appdomain.cloud:9000. For details, see [Network Requirements](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-client-requirementsnetwork-requirements).



curl https://<node>.securegateway.appdomain.cloud:9000
* If you are using proxy option, please check with your proxy team to ensure the Secure Gateway Client could access above host and port via the proxy.

curl -x <proxy> https://sgmanager.<region>.securegateway.cloud.ibm.com
curl -x <proxy> https://<node>.securegateway.appdomain.cloud:9000







 Secure Gateway Client connection errors 



* Initiate the failing request from the requesting application, check the Secure Gateway Client logs
* If the Secure Gateway Client is connecting to the tunnel but no client logs have been generated from the request, the issue is between the requesting application and the Secure Gateway Servers. This could range from network reliability to mismatched request protocols to an improper TLS mutual authentication handshake. For example:



* If the destination have enabled iptables rules, please ensure the IP of the cloud application is in the range of the iptables rules
* If the destination is set to use Mutual Auth, please ensure it is using associated certificate and key to connect



* If the client has generated error level logs from the request, then the issue is between the SG Client and the on-prem resource. For example:

[ERROR] Connection <connection ID> to destination <target host>:<target port> had error: '<Error Code>'
$ curl -X DELETE "https://us-south.kms.cloud.ibm.com/api/v2/keys/$KEY_ID" -H "authorization: Bearer $ACCESS_TOKEN" -H "bluemix-instance: $KP_INSTANCE_ID" -H "content-type: application/vnd.ibm.kms.key+json"
Show more



 JSON response 

{
"metadata": {
"collectionType": "application/vnd.ibm.kms.error+json",
"collectionTotal": 1
},
"resources": [
{
"errorMsg": "Conflict: 1 prior authorization(s) are required for deletion: Key could not be deleted. Please see reasons for more details.",
"reasons":
{
"code": "AUTHORIZATIONS_NOT_MET",
"message": "Number of authorizations required to delete is not met",
"status": 409,
"moreInfo": "https://cloud.ibm.com/apidocs/key-protect"
}
]
}
]
}











 15 - Only a single instance policy may be... 



 Message 

Only a single instance policy may be created per query parameter: Please pass single resource object

Reason code: NUM_COLLECTION_RESOURCE_ERR





 HTTP status code 

400 - Bad Request

The HTTP 400 Bad Request response status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).

The client should not repeat this request without modification.





 Context 

The create instance policy request fails because the same policy was specified more than once.

You may specify multiple policies in the request as long as each policy is unique. For example, you could create instance policies for dualAuthDelete and allowedIP in the same request.



 Example 

This request fails because more than one instance policy was provided.
File storage and block storage: Why does my PVC remain in a pending state? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

When you create a PVC and you run kubectl get pvc <pvc_name>, your PVC remains in a Pending state, even after waiting for some time.

 Why it’s happening 

During the PVC creation and binding, many different tasks are executed by the file and block storage plug-in. Each task can fail and cause a different error message.

 How to fix it 

Find the root cause by describing your PVC and reviewing the common error messages.



1. Describe the PVC and state.

kubectl describe pvc <pvc_name> -n <namespace>
2. Review common error message descriptions and resolutions.





File Storage for Classic error messages

 Error message Description Steps to resolve 

 User doesn't have permissions to create or manage StorageFailed to find any valid softlayer credentials in configuration fileStorage with the order ID %d could not be created after retrying for %d seconds.Unable to locate datacenter with name <datacenter_name>. The IAM API key or the IBM Cloud infrastructure API key that is stored in the storage-secret-store Kubernetes secret of your cluster does not have all the required permissions to provision persistent storage. See [PVC creation fails because of missing permissions](https://cloud.ibm.com/docs/containers?topic=containers-missing_permissions). 
 Your order will exceed the maximum number of storage volumes allowed. Please contact Sales Every IBM Cloud account is set up with a maximum number of file and block storage instances that can be created. By creating the PVC, you exceed the maximum number of storage instances. For more information about the maximum number of volumes that you can create and how to retrieve the number of volumes in your account, see the documentation for [file](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-managinglimits) and [block](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-managingstoragelimits) storage.
File storage and block storage: Why does my PVC remain in a pending state? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

When you create a PVC and you run oc get pvc <pvc_name>, your PVC remains in a Pending state, even after waiting for some time.

 Why it’s happening 

During the PVC creation and binding, many different tasks are executed by the file and block storage plug-in. Each task can fail and cause a different error message.

 How to fix it 

Find the root cause by describing your PVC and reviewing the common error messages.



1. Describe the PVC and state.

oc describe pvc <pvc_name> -n <project>
2. Review common error message descriptions and resolutions.





File Storage for Classic error messages

 Error message Description Steps to resolve 

 User doesn't have permissions to create or manage StorageFailed to find any valid softlayer credentials in configuration fileStorage with the order ID %d could not be created after retrying for %d seconds.Unable to locate datacenter with name <datacenter_name>. The IAM API key or the IBM Cloud infrastructure API key that is stored in the storage-secret-store Kubernetes secret of your cluster does not have all the required permissions to provision persistent storage. See [PVC creation fails because of missing permissions](https://cloud.ibm.com/docs/openshift?topic=openshift-missing_permissions). 
 Your order will exceed the maximum number of storage volumes allowed. Please contact Sales Every IBM Cloud account is set up with a maximum number of file and block storage instances that can be created. By creating the PVC, you exceed the maximum number of storage instances. For more information about the maximum number of volumes that you can create and how to retrieve the number of volumes in your account, see the documentation for [file](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-managinglimits) and [block](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-managingstoragelimits) storage.
2021/11/08 12:34:06 Request: RepoURL=https://github.ibm.com/wh-hp-insights/hi-cloud-automation, workspaceSource=Schematics, Branch=2021.10, Release=, Folder=terraform-v2/workspace-hi-qa-automation-app
2021/11/08 12:34:06 Related Activity: action=UPDATE_WORKSPACE,processedBy=sandbox-6bcf8bffcd-rxbww_2478
2021/11/08 12:34:06 Getting download command
2021/11/08 12:34:11 Fatal, could not download repo, Failed to clone git repository, couldn't find remote ref "refs/heads/2021.10" (most likely invalid branch name is passed)
2021/11/08 12:34:12 Problems found with the Repository. Please Rectify and Retry

If the Release parameter is empty and the Branch was set with release tag.

Schematics does not support release tag, as it's difficult to identify if it’s a release tag or a branch from the Git repository URL. You need to set the release tag through the [Schematics API](https://cloud.ibm.com/apidocs/schematics/schematicscreate-workspace).





 Why do I get a 403 error instead of 404 error when using an invalid workspace ID? 

curl -X GET https://schematics.cloud.ibm.com/v1/workspaces/badWOrkspaceId -H "Authorization: $IAM_TOKEN"
{"requestid":"3a3cbffe-e23a-4ccf-b764-042f7379c084","timestamp":"2021-11-11T17:00:07.169953698Z","messageid":"M1078","message":"Error while validating the location in the account. Verify you have permission to the location in the global catalog settings.","statuscode":403}
See [Creating a secret for the object storage service credentials](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-understandcreate_cos_secret). 
 object-path “<subdirectory_name>” not found inside bucket <bucket_name> You specified an existing subdirectory in your bucket that you want to mount to your PVC by using the ibm.io/object-path annotation. This subdirectory could not be found in the bucket that you specified. Verify that the subdirectory that you specified in ibm.io/object-path exists in the bucket that you specified in ibm.io/bucket. 
 BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. You set ibm.io/auto-create-bucket: true and specified a bucket name at the same time, or you specified a bucket name that already exists in IBM Cloud Object Storage. Bucket names must be unique across all service instances and regions in IBM Cloud Object Storage. Make sure that you set ibm.io/auto-create-bucket: false and that you provide a bucket name that is unique in IBM Cloud Object Storage. If you want to use the IBM Cloud Object Storage plug-in to automatically create a bucket name for you, set ibm.io/auto-create-bucket: true and ibm.io/bucket: "". Your bucket is created with a unique name in the format tmp-s3fs-xxxx. If you want to specify a bucket name for the auto created bucket, set ibm.io/auto-create-bucket: true and ibm.io/auto-delete-bucket: false and ibm.io/bucket: "<bucket_name>". 
 can't access bucket <bucket_name>: NotFound: Not Found You tried to access a bucket that you did not create, or the storage class and s3fs API endpoint that you specified don't match the storage class and s3fs API endpoint that were used when the bucket was created.
See [Creating a secret for the object storage service credentials](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understandcreate_cos_secret). 
 object-path “<subdirectory_name>” not found inside bucket <bucket_name> You specified an existing subdirectory in your bucket that you want to mount to your PVC by using the ibm.io/object-path annotation. This subdirectory could not be found in the bucket that you specified. Verify that the subdirectory that you specified in ibm.io/object-path exists in the bucket that you specified in ibm.io/bucket. 
 BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. You set ibm.io/auto-create-bucket: true and specified a bucket name at the same time, or you specified a bucket name that already exists in IBM Cloud Object Storage. Bucket names must be unique across all service instances and regions in IBM Cloud Object Storage. Make sure that you set ibm.io/auto-create-bucket: false and that you provide a bucket name that is unique in IBM Cloud Object Storage. If you want to use the IBM Cloud Object Storage plug-in to automatically create a bucket name for you, set ibm.io/auto-create-bucket: true and ibm.io/bucket: "". Your bucket is created with a unique name in the format tmp-s3fs-xxxx. If you want to specify a bucket name for the auto created bucket, set ibm.io/auto-create-bucket: true and ibm.io/auto-delete-bucket: false and ibm.io/bucket: "<bucket_name>". 
 can't access bucket <bucket_name>: NotFound: Not Found You tried to access a bucket that you did not create, or the storage class and s3fs API endpoint that you specified don't match the storage class and s3fs API endpoint that were used when the bucket was created.
Why can't I SSH into my host machines? 

  What’s happening 

When you try to SSH into a host machine that is assigned in Satellite, you see a message similar to the following.

Permission denied, please try again.

  Why it’s happening 

After the host is successfully assigned to a Satellite location control plane or cluster, Satellite disables the ability to SSH into the host for security purposes.

  How to fix it 

You cannot SSH into the host. If you need to modify settings on host machines in a cluster, try deploying a daemon set, such as in the [Tuning performance](https://cloud.ibm.com/docs/containers?topic=containers-kernel) example.

If you remove a host from your location or remove the entire location, you must reload the machine in your host infrastructure provider to SSH into the host again.
2021/11/08 12:34:06 ----- New Action -----
2021/11/08 12:34:06 Request: RepoURL=https://github.ibm.com/wh-hp-insights/hi-cloud-automation, workspaceSource=Schematics, Branch=2021.10, Release=, Folder=terraform-v2/workspace-hi-qa-automation-app
2021/11/08 12:34:06 Related Activity: action=UPDATE_WORKSPACE,processedBy=sandbox-6bcf8bffcd-rxbww_2478
2021/11/08 12:34:06 Getting download command
2021/11/08 12:34:11 Fatal, could not download repo, Failed to clone git repository, couldn't find remote ref "refs/heads/2021.10" (most likely invalid branch name is passed)
2021/11/08 12:34:12 Problems found with the Repository. Please Rectify and Retry

If the Release parameter is empty and the Branch was set with release tag.

Schematics does not support release tag, as it's difficult to identify if it’s a release tag or a branch from the Git repository URL. You need to set the release tag through the [Schematics API](https://cloud.ibm.com/apidocs/schematics/schematicscreate-workspace).
* Why do I get a 403 error instead of 404 error when using an invalid workspace ID?

curl -X GET https://schematics.cloud.ibm.com/v1/workspaces/badWOrkspaceId -H "Authorization: $IAM_TOKEN"
{"requestid":"3a3cbffe-e23a-4ccf-b764-042f7379c084","timestamp":"2021-11-11T17:00:07.169953698Z","messageid":"M1078","message":"Error while validating the location in the account.
Why does the Ingress status show an ESSSMG error? 

Virtual Private Cloud

Classic infrastructure

Satellite

  What’s happening 

When you check the status of your cluster's Ingress components by running the ibmcloud ks ingress status-report get command, you see an error similar to the following example.

Could not the secret group (ESSSMG).

  Why it’s happening 

IBM Cloud Kubernetes Service is unable to access the secret group that was registered with the cluster to upload the default Ingress certificates.

  How to fix it 

Review your service-to-service authorization policies and verify that communication between IBM Cloud Kubernetes Service and Secrets Manager is enabled.



1.  [Follow the steps to ensure there is a service-to-service authorization policy](https://cloud.ibm.com/docs/containers?topic=containers-secrets-mgrsecrets-mgr_setup_s2s) configured to enable communication between IBM Cloud Kubernetes Service and Secrets Manager.
2.  If the policy exists, verify that the secret group registered with the cluster exists in the instance.



*  To view the instance registration details for your cluster run the ibmcloud ks ingress instance ls command.
*  To view and modify the secret groups available in your instance, see [Organizing your secrets](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-secret-groups).
*  To update the secret group for your cluster, run the ibmcloud ks ingress instance default set command and specify the --secret-group option.



3.  If the issue persists, contact support. Open a [support case](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar). In the case details, be sure to include any relevant log files, error messages, or command outputs.