vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Overview 

The solution guide describes the tasks to create the two cyber-recovery solution architectures, Immutable backup and Isolated recovery environment. It also discusses the use cases for the creation of backups and then making this backup available to cyberadmins.

The solution guide uses:



* IBM Cloud® CLI commands to deploy the infrastructure.
* Ansible® for operating system configuration.
* PowerShell for the automation of Veeam®.



For a detailed understanding of the two solution architectures, see [Overview of cyber recovery with Veeam architecture](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-veeam-cr-sa-overview).



 Cyber-recovery immutable backup 



 Assumptions 



* The VMware vCenter Server® instance is already deployed and consists of three hosts in a consolidated cluster that runs VMware NSX-T™ and uses NFS data stores. Your vCenter Server instance might be different, and the step-by-step instructions can accommodate different instances.
* The Veeam service is already deployed and uses the bare metal option. However, the VSI instance is similar from a network perspective with two IP addresses; one on the primary subnet and one on the instance management subnet. The virtual machine (VM) option has a single IP address only on the instance management subnet.







 High-level steps 



* The [Planning the deployment](https://cloud.ibm.com/docs/vmwaresolutions/?topic=vmwaresolutions-veeam-cr-sag-planning) step captures and defines the information that is needed for the subsequent steps.
* [Provisioning the jump server](https://cloud.ibm.com/docs/vmwaresolutions/?topic=vmwaresolutions-veeam-cr-sag-jmp) and [Configuring the jump server](https://cloud.ibm.com/docs/vmwaresolutions/?topic=vmwaresolutions-veeam-cr-sag-jmpcfg) steps describe the tasks for the jump server.
This parameter is not used when you link to an existing repo. 
 repo_id optional, immutable String repo_id The ID of the GitLab project. 
 repo_name optional, immutable String repo_name The name of the GitLab repo to create. This parameter is required when you create a repo, or clone or fork a repo. This value is computed when you link to an existing repo. 
 repo_url optional, immutable String repo_url The URL of the GitLab repo for this tool integration. This parameter is required when you link to an existing repo. This value is computed when you create a repo, or clone or fork a repo. 
 source_repo_url optional, immutable String source_repo_url The URL of the repo that you want to fork or clone. This parameter is required when you fork or clone a repo, but it is not used when you create a repo or link to an existing repo. 
 token_url optional, updatable String token_url The token URL that is used to authorize with the GitLab server. 
 type required, immutable String type The operation to perform to initialize the new tool integration. Use new to create a Git repo, clone to clone an existing repo into a new Git repo, fork to fork an existing Git repo, or link to link to an existing Git repo. 







 Learn more about GitLab 

To learn more about GitLab, see the [GitLab article](https://www.ibm.com/garage/method/practices/code/tool_gitlab/) on the IBM Cloud Garage Method.
In the code snippet, Veeam port mapper service, mount acceptor port and NFS acceptor port are used by vPower NFS service that presents the backup files to VMware ESXi™ hosts. Veeam data transmission channels are used between Veeam components that include the Veeam Hardened backup repository.





 Adding the Linux hardened backup repository 

The following code snippet adds the Linux hardened repository as a Veeam managed server, and then adds the managed server as a backup repository.

- name: Connect to the local VBR server
win_shell: Connect-VBRServer

- name: Add the server as a Veeam Managed Server
win_shell: Add-VBRLinux -Name '{{ lhbr_fqdn }}' -SSHUser '{{ lhbr_user}}' -SSHPassword '{{ lhbr_password }}' -Description "Linux Hardened Repository - {{ lhbr_vbr_name }}" -SSHElevateToRoot:$true -SSHTempCredentials:$true -Confirm:$false

- name: Add the Veeam Managed Server as a Linux Hardened Backup Repository
win_shell: Add-VBRBackupRepository -Folder /mnt/veeamrepo01 -Type LinuxLocal -ImmutabilityPeriod '{{ immutability_period }}' -EnableBackupImmutability:$true -EnableXFSFastClone:$true -AlignDataBlocks:$true -UsePerVMFile:$true -MaxConcurrentJobs '{{ max_concurrent_jobs }}' -Server '{{ lhbr_fqdn }}' -Description "Linux Hardened Repository" -Name '{{ lhbr_vbr_name }}'



* -SSHTempCredentials:$true is used so that the SSH credentials are not entered into Veeam's credential store.
* -EnableXFSFastClone:$true defines that the Fast Clone technology is enabled for the backup repository.
Using tethered endpoints 

When deciding how to configure your IBM Cloud® Object Storage instance, consider how the endpoints reflect your needs for resiliency and access.

When you use a Cross Region bucket, it is possible to direct your accesses to a tethered endpoint associated with a specific Cross Region metropolitan area, rather than connecting to the nearest available Cross Region metropolitan area. In contrast to the geo endpoint, when you send requests to a tethered end point there is no automated failover if that region becomes unavailable. Applications that direct traffic to a tethered endpoint must implement appropriate failover logic internally to achieve the availability advantages of the Cross Region storage.

One reason for using a tethered endpoint is to control where data ingress and egress occurs while still distributing the data across the widest possible area. Imagine an application running in the us-south region that wants to store data in a US cross-region bucket but wants to ensure that all read and write requests remain in the Dallas area:



1. The application creates a client using the https://s3.private.dal.us.cloud-object-storage.appdomain.cloud endpoint.
2. The Object Storage service in Dallas suffers an outage.
3. The application detects a persistent failure trying to use the tethered endpoint.
4. The application recognizes the need to fail over to a different tethered endpoint, such as San Jose.
5. The application creates a new client using the https://s3.private.sjc.us.cloud-object-storage.appdomain.cloud endpoint.
6. Connectivity is resumed, and access can be re-routed to Dallas when service is restored.



When sending requests to a tethered endpoint there is no automated failover if that region becomes unavailable.

For contrast, imagine another application using the normal US cross-region endpoint:



1. The application creates a client using the https://s3.us.cloud-object-storage.appdomain.cloud endpoint.
2. The Object Storage service in Dallas suffers an outage.
3.
: ODF uses these devices to create a virtualized storage layer, where your app data is replicated for high availability. Because ODF abstracts your underlying storage, you can use ODF to create File, Block, or Object storage claims from the same underlying raw block storage.

For a full overview of the features and benefits, see [OpenShift Data Foundation](https://www.redhat.com/en/technologies/cloud-computing/openshift-data-foundation).



 Architecture overview 

Review the following diagram and table to learn more about OpenShift Data Foundation.

Zoom

![ODF architecture](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/ODF_components.svg)

Figure 1. ODF architecture



ODF architecture overview

 Number ODF component Description 

 1 OpenShift Data Foundation storage classes When you deploy ODF, the ODF operator creates File, Block, and Object storage classes in your cluster. Reference these storage classes in your PVCs and to claim storage for your apps. 
 2 Object Storage Device (OSD) These devices provide application storage in your cluster. Each OSD is a raw block storage device that can be a local disk on your worker node or dynamically provisioned when you deploy ODF. In VPC clusters, your OSD block storage devices are dynamically provisioned by using the Block Storage for VPC driver. In Satellite clusters, you can use local volumes on your worker nodes, or dynamically provision block storage devices by using a block storage driver that supports dynamic provisioning. In Classic clusters, the OSD block devices are local disks on your worker nodes. When you deploy ODF, each device is mounted by an OSD pod. The total storage that is available to your applications is equal to the osdSize multiplied by the numOfOsd. 
 3 Object Storage Daemon (OSD) pods The OSD pods manage data placement and replication across your storage devices.
app.set('view engine', 'ejs');
app.use(bodyParser.json());

var title = 'COS Image Gallery Web Application';
// Serve index.ejs
app.get('/', function (req, res) {
res.render('index', {status: '', title: title});
});

//...

The following figure shows what the index view template when rendered and sent to the browser. If you are using ,nodemon you might have noticed that your browser refreshed when you saved your changes.

Zoom

![uploadimageview](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/web-app-tutorial-018-templates.jpg)

Figure 8. Your updated web app by using templates and views for displays

Our view templates share HTML code between the <head>...</head>; tags, so we placed it into a separate include template. This template (head-inc.ejs) contains a scriptlet (a binding for a JavaScript variable) for the page title on line 1. The title variable is set in app.js, and passed in as data for our view template in the line below that. Otherwise, we are simply using some CDN addresses to pull in Bootstrap CSS, Bootstrap JavaScript, and JQuery. Finally, we add a custom static styles.css file from our pubic/stylesheets directory.

<title><%=title%></title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
One of true, false.
* logpull_options: The configuration string. For example, timestamps=rfc3339&timestamps=rfc3339
* dataset: The dataset that is pulled. One of http_requests, range_events, firewall_events.
* frequency: The frequency at which CIS sends batches of logs to your destination. One of high, low.





3. When all variables are initiated, create the logpush job:



* Log Analysis example

{
"logdna": {
"hostname": "example.com",
"ingress_key": "",
"region": "us-east"
},
"dataset": "range_events",
"enabled": false,
"name": "CIS-Range-LogDNA",
"frequency": "low",
"logpull_options": "fields=RayID,ZoneID&timestamps=rfc3339"
}
* COS example

{
"cos": {
"bucket_name": "example_bucket",
"path": "temp/",
"id": "cos_instance_id",
"region": "us-east"
},
"dataset": "firewall_events",
"enabled": false,
"name": "CIS-Firewall-COS",
"frequency": "low",
"logpull_options": "fields=RayID,ZoneID&timestamps=rfc3339",
"ownership_challenge": "xxxxxxx"
}
{LOGDNA_REGION}.logging.cloud.ibm.com/logs/ingest?hostname={DOMAIN_NAME}&apikey={LOGDNA_INGESTION_KEY}). For more information, see [Log Analysis regions](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-regions).
* --name specifies the Logpush job name.
* --fields specifies the list of log fields to be included in log files. Use commas to separate multiple fields.
* --enable is the flag to enable or disable the Logpush job. Valid values are true or false (default).
* --dataset is the dataset that is pulled. One of http_requests, range_events, firewall_events.
* --frequency is the frequency at which CIS sends batches of logs to your destination. One of high, low.







 Creating Logpush with COS CLI 

To create a Logpush job for a specific domain and enable the job using COS, run the following command:

ibmcloud cis logpush-job-create DNS_DOMAIN_ID --destination BUCKET_PATH --name JOB_NAME --fields all --enable true

Where:



* --destination specifies the path to the Object Storage bucket.

It follows the syntax: cos://<bucket_path>?region=xxx&instance-id=xxxx, where bucket_path is the bucket name followed by an optional path-like structure, region and Object Storage instance-id are the Object Storage bucket region and instance ID, which are required arguments.

For example, cos://mybucket/cislog?region=us-south&instance-id=c84e2a79-ce6d-3c79-a7e4-7e7ab3054cfe.
* --name specifies the Logpush job name.
* --enable is the flag to enable or disable the Logpush job. Valid values are true or false (default).
* --fields specifies the list of log fields to be included in log files.
Example 

ibmcloud fn property unset --namespace









 Rule commands 

Create, delete, enable, disable, update, and find information about rules.

To see CLI help for the rule command, run ibmcloud fn rule.



 ibmcloud fn rule create 

Create a rule to associate a trigger with an action. Before you can create a rule, create a trigger and an action first.

ibmcloud fn rule create RULE_NAME TRIGGER_NAME ACTION_NAME



 Example 

ibmcloud fn rule create myrule mytrigger myaction

Output

ok: created myrule







 ibmcloud fn rule delete 

To clean up your namespace, remove rules you no longer need.

ibmcloud fn rule delete RULE_NAME [--disable]



 Command options 

RULE_NAME
: The name of a rule. This value is required.

--disable
: Disable the rule before you delete it.





 Example 

ibmcloud fn rule delete myrule







 ibmcloud fn rule disable 

Change the status of a rule to inactive and stop it from running an action when a trigger is fired.

ibmcloud fn rule disable RULE_NAME



 Example 

ibmcloud fn rule disable myrule







 ibmcloud fn rule enable 

Change the status of a rule from inactive to active. When active, an action runs when a trigger is fired.

ibmcloud fn rule enable RULE_NAME



 Example 

ibmcloud fn rule enable myrule







 ibmcloud fn rule get 

Get metadata that describes a specific rule.

ibmcloud fn rule get RULE_NAME [--summary]



 Command options 

RULE_NAME
: The name of a rule. This value is required.

--summary
: Get a summary of the rule details.





 Example 

ibmcloud fn rule get myrule







 ibmcloud fn rule list 

List all the rules that you created or a specific number of rules.
The MongoDB database has three pods. It runs in replicaSet mode, which means that one pod runs in the coordinator role in read/write mode, and the rest of the pods run in the secondary role and are read-only. Changes from the coordinator pod are replicated to the secondary pods. During the service installation, data is loaded into the Mongo database by a Kubernetes job. No data is written to the Mongo database after the service is installed. Only the Recommends microservices read data from Mongo. On creation, Recommends pods check whether Mongo is running, and wait until the Mongo database is loaded with the required data.

The process of loading data for Recommends into the Mongo database that happens as part of the installation can take 30 minutes (or more).



* The mongoDB pod names follow the convention ${release-name}-ibm-mongodb-server-[0-9]. For longer release names, the name is shorten to ${release-name prefix}-[a-f0-9]{4}-336f-server-.
* The names of the Kubernetes job that runs during the installation of the service is ${release-name}-recommends-load-mongo.



The ${release-name} is watson-assistant.



4.7.0: The following data source is no longer used, starting with the 4.7.0 release:



* MinIO: MinIO is an object storage service that implements the Amazon S3 API. It is used by the language understanding pipeline microservices (NLU, Master, TAS, ED-MM, and the training pods) to store and load trained models for intent and entity classification. Data is stored in the nlclassifier-icp bucket. In Watson Assistant, MinIO is often referred to as COS, which stands for Cloud Object Storage. For more information, see [MinIO](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-architecturearchitecture-minio).





 Architecture changes 



* 1.5.0: The following changes to the architecture occurred with this release:
Cognos Dashboard Embedded end user experience 

IBM® Cognos Dashboard Embedded uses the powerful IBM Cognos Analytics dashboard experience. To learn more above the Cognos Analytics dashboard experience, please see [IBM Cognos Analytics - Dashboard documentation](https://www.ibm.com/support/knowledgecenter/en/SSEP7J_11.0.0/com.ibm.swg.ba.cognos.ug_ca_dshb.doc/wa_dashboard_discoveryset_intro.html).

Cognos Dashboard Embedded does not support:



*  Reporting and modelling.
*  Custom widgets. You cannot extend the set of shapes.
*  Pinned objects. There is no My pins panel.
*  Intent based authoring.
*  Drillthrough navigation.
Overview of Cognos Dashboard Embedded 

IBM® Cognos Dashboard Embedded provides developers the ability to embed a visualization platform directly into their application.

If instead of embedding dashboards in your applications with APIs, you want to create visualization in a dashboard editor, see [Cognos Dashboards on Cloud Pak for Data as a Service](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/analytics-dashboard.html).

![dashboard_intro](https://cloud.ibm.com/docs-content/v1/content/2781097afe16f2d329d7a606ddd260fb1a3e355c/services/cognos-dashboard-embedded/dashboard_intro.jpg)

You as a developer have the flexibility to define the user workflow and control the options available to users. You can choose from a guided exploration of the analysis through authored fixed dashboards, to a free-form analytic exploration environment in which users choose their own visualizations – and virtually anything in between.

Complete these four steps to add Cognos Dashboard Embedded into an application:



1.  Creating an IBM Cloud® account and provisioning a Cognos Dashboard Embedded service instance.
2.  Creating a service credential to access the service instance API.
3.  Creating a Cognos Dashboard Embedded session from your web service application.
4.  Adding Cognos Dashboard Embedded into your web client application through the Cognos Dashboard Embedded JavaScript API.
5.  Cognos Dashboard Embedded uses the powerful IBM Cognos Analytics dashboard experience. To learn more above the Cognos Analytics dashboard experience, please see [IBM Cognos Analytics - Dashboard documentation](https://www.ibm.com/support/knowledgecenter/en/SSEP7J_11.1.0/com.ibm.swg.ba.cognos.ug_ca_dshb.doc/wa_dashboard_discoveryset_intro.html).



For more information, see [Getting started tutorial](https://cloud.ibm.com/docs/services/cognos-dashboard-embedded/dde_getting_started.html).
ibmcloud cis logpull 31984fea73a15b45779fa0df4ef62f9b --start 2020-05-18T12:14:58Z --end 2020-05-18T13:14:58Z --fields ClientIP,EdgeServerIP,ClientRequestHost --count 10 --sample 1 --timestamps rfc3339 --output JSON
<-- </section "id="section-examples-logpull" "> --><-- </section "id="section-cli-logpull" "> --><-- </section "id="section-logpull-section" "> --><-- <section "id="section-log-push-cli-ref" "> --> Log push (Enterprise plan only) <-- <section "id="section-logpush-job-create" "> --> ibmcloud cis logpush-job-create Create a new log push job for a given domain. Before using this command grant write access to your IBM Cloud Object Storage bucket to the IBM Cloud account cislogp@us.ibm.com]cislogp_us_.ibm_.com] . (Enterprise plan only) ibmcloud cis logpush-job-create DNS_DOMAIN_ID --destination DESTINATION_URL --name NAME --enable true|false] --fields FIELDS | all] --timestamps format]--dataset DATASET] --frequency FREQUENCY] -i, --instance INSTANCE] --output FORMAT] ! !! ] ! ! ! ! !
<-- <section "id="section-command-options-logpush-job-create" "> --> Command options DNS_DOMAIN_ID: The ID of DNS domain. Required.--destination value: Specify a COS bucket path or a LogDNA path where data will be pushed.
<-- <ul> -->
A Logpush job is created successfully after CIS validates the ownership challenge. The Logpush job pushes request logs to your Object Storage bucket every 30 seconds or every 100,000 records, whichever comes first. More than one file might be pushed per 30-second period or per 100,000 records.

Logpush jobs created prior to September 2020 might continue pushing every 5 minutes. Any modification to one of these older jobs triggers an update to the push frequency.

You can use the token {DATE} in the bucket path to make the Logpush job push request logs in daily folders in the bucket path. For example: cos://mybucket/cislog/{DATE}?region=us-south&instance-id=c84e2a79-ce6d-3c79-a7e4-7e7ab3054cfe







 Setting up Logpush using the API 

To create a logpush job using the API, take the following steps:



1. Set up your API environment with the correct variables.
2. Store the following values in variables to be used in the API command:



* crn: the full url-encoded CRN of the service instance.
* zone_id: the domain ID.
* request body: information to create the logpush job body. One of logpush_job_cos_req, logpush_job_logdna_req.



* For COS, cos: information to identify the COS bucket where the data is pushed.
* For Log Analysis, logdna: information to identify the Log Analysis instance where the data is pushed.
* name: The name of the logpush job.
* enabled: Whether the job is enabled. One of true, false.
* logpull_options: The configuration string. For example, timestamps=rfc3339&timestamps=rfc3339
* dataset: The dataset that is pulled. One of http_requests, range_events, firewall_events.
* frequency: The frequency at which CIS sends batches of logs to your destination. One of high, low.





3.
x-amz-request-id: da214d69-1999-4461-a130-81ba33c484a6
ETag: "37d4c94839ee181a2224d6242176c4b5"
Content-Type: text/plain; charset=UTF-8
Last-Modified: Thu, 25 Aug 2016 17:49:06 GMT
Content-Length: 11

--------------------





 Download an object 

A GET given a path to an object downloads the object.

The Etag value that is returned for objects encrypted using SSE-C/SSE-KP will not be the MD5 hash of the original decrypted object.

Not all operations are supported in Satellite environments. For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)

Syntax

GET https://{endpoint}/{bucket-name}/{object-name} path style
GET https://{bucket-name}.{endpoint}/{object-name} virtual host style





 Delete an object 

A DELETE given a path to an object deletes an object.

Not all operations are supported in Satellite environments. For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)

Syntax

DELETE https://{endpoint}/{bucket-name}/{object-name} path style
DELETE https://{bucket-name}.{endpoint}/{object-name} virtual host style

Example request

DELETE /apiary/soldier-bee HTTP/1.1
Authorization: Bearer {token}