vania-janet/multiturn-rag-retrieval-data huggingface dataset:
NAME Version Health State Health Status
istio 1.5 normal Addon Ready
3. Check if there are any operators in use that were deployed through OLM by listing the cluster service versions.

kubectl get clusterserviceversions -A

Example output of an operator that was deployed through OLM.

NAMESPACE NAME DISPLAY VERSION REPLACES PHASE
kubernetes-operator-lifecycle-manager packageserver Package Server 0.19.0 Succeeded

If there are clusterserviceversions listed in the output, you are still using operators that are managed through OLM. You can continue to use the OLM, but you are responsible managing updates. The following steps assume you are no longer using OLM and provide clean up instructions, do not continue if you are still using OLM.
4. If there were no clusterserviceversions instances in the output of step 3, you are not using OLM. You can now get a list of the resources that were created by OLM and remove them.
5. For each CRD in the following list, get the custom resources (CRs) across all namespaces.

kubectl get catalogsources.operators.coreos.com -A
kubectl get clusterserviceversions.operators.coreos.com -A
kubectl get installplans.operators.coreos.com -A
kubectl get operatorgroups.operators.coreos.com -A
kubectl get operators.operators.coreos.com -A
kubectl get subscriptions.operators.coreos.com -A
6. If you find CRs, determine whether you still need these resources.



* You might still see the olm-operators and ibm-operators operator groups that were deployed by OLM. These operator groups were part of the installation of OLM. Because you do not have any cluseterserviceversions instances in the previous steps, you are no longer using these operator groups.
Figure 1. Architecture diagram of the tutorial



1. Application generates log events to Event Streams.
2. To persist the log events, they are stream landed into Object Storage through Data Engine.
3. The storage bucket and the Data Engine jobs are encrypted with Key Protect service. Also, the stream landing job executes in Data Engine by securely retrieving the service ID from Key Protect.
4. Auditor or support staff use Data Engine or Analytics Engine to perform requests.
5. Requests are executed against the data stored in Object Storage.







 Before you begin 

This tutorial requires:



* IBM Cloud CLI,
* IBM Cloud CLI plugin: ibmcloud plugin install analytics-engine-v3
* git to clone a code repository that contains some example data.
* Optionally, a Docker client.



You will find instructions to download and install these tools for your operating environment in the [Getting started with solution tutorials](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-tutorials) guide.





 Step 1: Create services 

In this section, you will create the services required to perform analysis of log events generated by your applications. Choose a region that supports all services.

The Analytics Engine serverless plan has limited availability. Dallas, us-south, was verified for this tutorial.



 Enable Platform Logs 

Platform logs are generally useful for troubleshooting resources and will be required in a later step to see the output of Analytics Engine.

You can have multiple IBM Cloud Log Analysis instances, however, only one instance in a region can be configured to receive platform logs from [enabled cloud services](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-cloud_services) in that IBM Cloud region.



1. Navigate to the [Observability](https://cloud.ibm.com/observe) page and click Logging, look for any existing log analysis services with Platform logs enabled. If there is a platform logging instance in the region no further configuration is required.
2. To create a new IBM Cloud Log Analysis click the Options then Create.
Set the API endpoint for your region and log in:

ibmcloud api https://cloud.ibm.com
ibmcloud login
3. Get the list of the resource groups for your account and select one of the returned resource group as the target resource group in which to create the IBM Analytics Engine serverless instance:

ibmcloud resource groups
ibmcloud target -g <resource_group_name>
4. Create a service instance:

ibmcloud resource service-instance-create <service_instance_name> ibmanalyticsengine <plan_name> <region> -p @<path_to JSON file with cluster parameters>

For example, for the Dallas region:

ibmcloud resource service-instance-create MyServiceInstance ibmanalyticsengine standard-serverless-spark us-south -p @provision.json

You can give the service instance any name you choose. Note that currently, standard-serverless-spark is the only supported serverless plan and us-south and eu-de the only supported regions.

The provision.json file contains the provisioning parameters for the instance you want to create.

The endpoint to your IBM Cloud Object Storage instance in the payload JSON file should be the direct endpoint. Direct endpoints provide better performance than public endpoints and do not incur charges for any outgoing or incoming bandwidth.

This is a sample of what the provision.json file can look like. See [Architecture and concepts in serverless instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-concepts) for a description of the provisioning parameters in the payload.

Note that both Spark 3.1 and Spark 3.3 are supported. If you don't specify a default Spark runtime version when you create a service instance, Spark 3.1 is taken by default.

{
"default_runtime": {
"spark_version": "3.1"
},
"instance_home": {
"region": "us-south",
Architecture for a resilient infrastructure for a single availability zone 

Deploying your application across the availability zone in a multiple-zone region provides the highest resiliency, 4 9's. However, there are scenarios where deploying across the zones is not wanted, such as needing the latency to be minimal or the application is deemed not critical.



 Virtual server instance Consideration 

Whatever the case might be, there are a few things to consider when you are designing for resiliency for a single zone. A single virtual server instance provides a 3 9's, but deploying multiple virtual server instances of the common workload can possibly increase it to 99.95%. It is recommended to deploy at least 3, even if only 2 are deployed. Placement groups should be used when you deploy multiple virtual server instances. While the IBM Cloud® VSI scheduler attempts to place the virtual server instances on different hypervisors, the scheduler might not succeed. If they are not, then a single point of failure is introduced because the virtual server instances are placed on the same hypervisor.

By creating multiple virtual server instances that offer the same service and use placement groups, the application is better protected against hardware or software failures, and even for unplanned maintenance.

If the group of virtual server instances in the tier is less than 4, then use power spread. Anything greater than 4, use hypervisor spread.

For more information about placement groups, see [About placement groups](https://cloud.ibm.com/docs/vpc?topic=vpc-about-placement-groups-for-vpc).

Scaling should also be considered. While scaling may not directly impact the overall resiliency, it does impact the application availability where the virtual server instance resources are exhausted and are no longer able to handle the increased demand.

Scaling can be done multiple ways. For example, scaling up, where a higher profile is being provisioned from either a snapshot volume or building a new system. Another example, and probably the most effective, is using the Auto scale feature for virtual server instance. Auto scale can add or remove virtual server instances based on the current demand.
It maps the logical locks to the database.





Db2 for standard system



* Database instance (DB) - Db2 in this case. For more information, see [AnyDB - IBM Db2](https://cloud.ibm.com/docs/sap?topic=sap-anydb-ibm-db2) and [Infrastructure certified for SAP](https://cloud.ibm.com/docs/sap?topic=sap-iaas-offerings).
* Primary application server instance (PAS instance) - The global directories of the ASCS instance can be used as the global file system. That means that the host with the ASCS instance is the SAP global host. However, you can also separately install the global directories on any host of your SAP system landscape. You can also use the SAP transport host or the host with the global file system (SAP global host) as your primary application server instance host. Optionally, you can install one or more extra application server instances.
* Additional Application Server (AAS) - You can install one or more extra application server instances for an existing SAP system. Additional application server instances are optional and can be installed on separate hosts.

An extra application server instance can run on:



* The host of any instance of the existing SAP system
* On a dedicated host









 Distributed system 

In a distributed system, there are multiple virtual server instances and every instance can run on a separate host:

Zoom

![Figure 3. Distribured installation](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/refarch-sap-db2-dist-only.svg)

Figure 3. SAP NetWeaver 7.x with Db2 distributed installation with AAS

The components in a distributed system are the same as the components in a standard system, but there are restrictions as to which instances can go on which hosts.







 Related information 

SAP One Support Notes that apply to this document:
See [Configuring application scaling](https://cloud.ibm.com/docs/codeengine?topic=codeengine-app-scale) and be aware of [Application defaults and limits](https://cloud.ibm.com/docs/codeengine?topic=codeengine-limitslimits_application).



Table 3: Number of pods that the autoscaler requested

 Metadata Description 

 Metric Name ibm_codeengine_application_requested_instances 
 Metric Type gauge 
 Value Type none 
 Segment By Service instance, name of the namespace, project name, application name, application revision name 







 ibm_codeengine_application_not_ready_instances 

Number of pods that are not ready currently.

Use this metric to observe if there are issues with your running application.

Application instances that are in not ready state cannot serve requests. This not ready state indicates that something is preventing the app from becoming ready.



Table 4: Number of pods that are not ready currently

 Metadata Description 

 Metric Name ibm_codeengine_application_not_ready_instances 
 Metric Type gauge 
 Value Type none 
 Segment By Service instance, name of the namespace, project name, application name, application revision name 







 ibm_codeengine_application_pending_instances 

Number of pods that are pending currently.

Use this metric to observe if there are issues with your running application.

You might observe pending applications when you observe automatic application scaling. Application instances that are pending are waiting to be scheduled. If your application remains in a pending state, perhaps the application cannot start because of insufficient resources, such as memory or CPU. See [Supported memory and CPU combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo).



Table 5: Number of pods that are pending currently

 Metadata Description 

 Metric Name ibm_codeengine_application_pending_instances 
 Metric Type gauge 
 Value Type none 
 Segment By Service instance, name of the namespace, project name, application name, application revision name 







 ibm_codeengine_application_desired_instances 

Number of pods that the autoscaler wants to allocate.
If you have a 1U single socket server in one of these sites, you might want to consider a 2U chassis with redundant power supplies. For more information about availability zones, see [Locations for resource deployment](https://cloud.ibm.com/docs/overview?topic=overview-locations).





 Add on features for highly available compute resources 

At a minimum, deploying more than one virtual server instance helps make your solution highly available. IBM Cloud has more features that you can add to your solution:



* Placement groups that distribute your VSIs on different hypervisors or on different hypervisors with different power sources.
* Auto scale that aoutmtically scales your number of VSIs up and down based on capacity.





 Placement groups 

You can use [placement groups](https://cloud.ibm.com/docs/vpc?topic=vpc-about-placement-groups-for-vpc) to ensure that multiple virtual server instances are on different physical servers within a data center or availability zone. Placing virtual server instances on different physical servers improves your overall redundancy and availability story. You can improve your redundancy more by spreading placement groups across different availability zones. Your business applications continue to run and provide service to your customers even if one or more virtual server instances are lost.

You have 2 different placement algorithms available to choose from:



* Hypervisor spread - Ensures the VSIs are distributed across different hypervisors. There can be up to 12 VSIs in the placement group.
* Power spread - Ensures the VSIs are placed not only different hypervisors but also the hypervisors are placed on different power and network, essentially on different racks. There can be up to 4 VSIs in the placement group.



Zoom

![Placement Groups.](https://cloud.ibm.com/docs-content/v1/content/df269b100c3499f1efcbb7d030dce8dec25c539b/ha-infrastructure/images/placement_groups.svg)

Figure 6. Placement group algorithms





 Auto scale
Scaling, high availability, and traffic management 

When your app is deployed to Code Engine, it is accessible and available for tests and even production. Because Code Engine provides a serverless platform for containerized workloads, most of your runtime details are managed automatically by Code Engine. You can, however, control many of the details.



 Scaling your app 

When working with Cloud Foundry, you might have used the [autoscaling feature](https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-autoscale_cloud_foundry_apps). With this feature, you can automatically add and remove app instances, depending on performance metrics or date and time. By default, autoscaling is set to off, and some setup work is required to use it. In contrast, Code Engine autoscales your application by default. Code Engine scales an app up to 10 runtime instances (by default) and down to zero (meaning that the app does not consume any resources when not in use).

You can edit these settings within the [limits and quotas of the Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-limits). A max scale of 0 (zero) scales your app to the maximum possible, if necessary while a min scale of 1 keeps at least one instance of your application running. By adjusting the concurrency and request timeout, you can tune the autoscaler to your needs. For more information, see [Configuring application scaling](https://cloud.ibm.com/docs/codeengine?topic=codeengine-app-scale)





 High availability 

For aspects of high availability (and disaster recovery), Cloud Foundry and Code Engine are similar.



* Follow best practices for handling concurrent requests.
* Make sure to have at least two or even three instances of the app up and running.
* Run app instances in the available zones of a multi-zone region (MZR).
There are various ways an operational logging solution can be implemented. See [Setting up an operational logging solution](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-logging-operational-tutorial) for one example that uses the Red Hat OpenShift on IBM Cloud Container Platform Elasticsearch, Fluentd, and Kibana EFK stack.





 Platform logging 



 VPC network traffic 

If you are using virtual server instances, you are required to use [Flow Logs for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-flow-logs) to enable the collection, storage, and presentation of information about the Internet Protocol (IP) traffic going to and from network interfaces on virtual server instances within your VPC.

Flow logs can help with a number of tasks, including:



* Troubleshooting why specific traffic isn't reaching an instance, which helps to diagnose restrictive security group rules
* Recording the metadata network traffic that is reaching your instance
* Determining source and destination traffic from the network interfaces
* Adhering to compliance regulations
* Assisting with root cause analysis



See [Creating a flow log collector](https://cloud.ibm.com/docs/vpc?topic=vpc-ordering-flow-log-collector) for more details. You must configure flow logs to go to a properly secured IBM Cloud Object Storage bucket, [encrypted with KYOK](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-rest).





 Red Hat OpenShift on IBM Cloud and virtual server instances 

Platform logs generated from Red Hat OpenShift on IBM Cloud and virtual server instances can be collected by using your own software solution or by using the previously mentioned solution for [Setting up an operational logging solution](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-logging-operational-tutorial).





 Other IBM Cloud services 

There is not a Financial Services Validated solution for capturing operational platform logs from other IBM Cloud services.
CP-8 (2) - Single Points of Failure 



  Control requirements 

CP-8 (2) - 0
:   The organization obtains alternate telecommunications services to reduce the likelihood of sharing a single point of failure with primary telecommunications services.





  IBM Cloud for Financial Services profile 

The rules related to this control that follow are part of the IBM Cloud for Financial Services v1.2.0 profile in [IBM Cloud® Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-getting-started).



*  Check whether there are at least # instances of Direct Link in an account
Batch job workloads 

IBM Cloud® Code Engine is a fully managed, serverless platform that runs your containerized workloads, including batch job or application workloads. Learn about running batch jobs in IBM Cloud® Code Engine.



 What are batch job workloads? 

A job in Code Engine runs one or more instances of your executable code. Unlike applications, which handle HTTP requests, jobs are designed to run one time and exit, such that the resources to run the job workload are freed up.

You can scale batch jobs by defining multiple instances. Workloads can be split into parallel tasks to reduce compute time. You can set the job to run with automatic retry to address failing workloads within a job instance. You can trigger batch jobs manually, programmatically, and by events, such as Object Storage events.

When you create a job, you can specify workload configuration information that is used each time that the job is run.

Typical batch job workloads include,



* machine model training
* analyzing files, such as voice analysis or image recognition
* compressing or decompressing files
* archiving information



When you create a job, you can specify workload configuration information that is used each time that the job is run.



 What is the lifecycle of a batch job? 

When you submit a batch job, it runs to completion. Typically, batch jobs retrieve input data, do computational work, and store the results in persistent data stores. When the batch job is completed, resources that are used to run the job are removed and no cost is incurred for any stand-by resources.







 How do jobs compare to apps and functions? 



Table 1. Comparing Code Engine apps, jobs, and functions

 Characteristic App Job Function 

 Execution time (duration) Long-running 10min per request Long-running up to 24hrs Short-running (<=2mins) 
 Startup latency Medium Scheduled start Low 
 Termination Run-continuously Run-to-completion Run-to-completion 
 Invocation On request or permanently running Scheduled On request, instant 
 Programming Model Container based build and execution Container based build and execution Language specific source code files and dependency meta data 
 Parallelism Parallel execution, flexible Low to medium parallel execution High parallel execution
IBM Cloud services that generate IBM Log Analysis logs by location 

List of locations where IBM Cloud services are enabled to send logs to IBM® Log Analysis. You monitor these logs that you monitor through the Log Analysis instance that is configured to receive platform services logs. [Learn more about enabling service platform logs](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logs).



 Analytics services 

The following table list the locations where automatic collection of analytics service logs is enabled. You can monitor logs through the Log Analysis instance that is available in the same location as your analytics service instance.

America

Asia Pacific

Europe



Table 1. Analytics services

 Service Dallas (us-south) Washington (us-east) 

 Planning Analytics as a Service ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/checkmark-icon.svg) 







 Compute serverless services 

America

Asia Pacific

Europe



Table 2. Compute serverless services integration in America locations

 Service Dallas (us-south) Washington (us-east) 

 Cloud Functions ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/checkmark-icon.svg) 
 Code Engine !
IBM Cloud platform services that send metrics to IBM Cloud Metrics Routing

 Service CRN service name Metrics 

 [IBM Cloud Activity Tracker Event Routing](https://cloud.ibm.com/docs/atracker) atracker 
 [IBM Cloud Metrics Routing](https://cloud.ibm.com/docs/metrics-router) metrics-router 







 Security services 



Table 7. List of security services

 Service CRN service name More info 

 [IBM® Key Protect](https://cloud.ibm.com/docs/key-protect?topic=key-protect-getting-started-tutorialgetting-started-tutorial) kms [Platform metrics](https://cloud.ibm.com/docs/key-protect?topic=key-protect-operational-metrics) 
 [IBM Cloud Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-get-started) hs-crypto [Platform metrics](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-operational-metrics) 







 Serverless 



Table 8. List of serverless services

 Service CRN service name Metrics 

 [IBM Cloud Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-getting-started) codeengine [Platform metrics](https://cloud.ibm.com/docs/codeengine?topic=codeengine-monitor) 







 VMware 

The following table lists Cloud services that are enabled for IBM Cloud Monitoring:



Table 9. VMware

 Service CRN service name Metrics 

 [VMware as a Service](https://cloud.ibm.com/docs/vmware-service?topic=vmware-service-getting-started) vmware [Platform metrics](https://cloud.ibm.com/docs/vmware-service?topic=vmware-service-single-tenant-monitoring)