vania-janet/multiturn-rag-retrieval-data huggingface dataset:
SI-12 - Information Handling and Retention 



  Control requirements 

SI-12 - 0
:   The organization handles and retains information within the information system and information output from the system in accordance with applicable federal laws, Executive Orders, directives, policies, regulations, standards, and operational requirements.





  Additional IBM Cloud for Financial Services specifications 

The organization shall adhere to data retention and destruction parameters defined by the customer.

Destruction of customer records must be approved by the customer prior to destruction.

Destruction of customer data must occur within 30 days of customer authorization.

The organization will inform the customer upon completion of requested/approved modification/disposal of customer data.

The organization will ensure that metadata for records owned by FS-ready public cloud customer but maintained by the organization will meet customer granularity requirements. These requirements can be requested from the customer assigned Vendor Manager to the organization/IBM.

Rules of behavior must include data handling requirements according to customer’s security classification, including but not limited to:



*  Clear desk policy to safeguard sensitive information
*  Customer data may not be stored on laptops or mobile devices
*  When users stop work and move away from the immediate vicinity of the system, screen locks must be used to conceal information previously visible on the display with a publicly viewable image
*  Credentials must not be shared and must be rotated and stored in accordance with authentication and encryption policies
*  Collaboration spaces must be secured and data only shared with authorized individuals granted permissions to access the data.







  IBM Cloud for Financial Services profile 

The rules related to this control that follow are part of the IBM Cloud for Financial Services v1.2.0 profile in [IBM Cloud® Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-getting-started).



*  Check whether Cloud Object Storage quota enforcement is off for buckets that are configured to use Activity Tracker Event Routing
*  Check that Hyper Protect Crypto Services has failover units in at least 2 different regions that are Financial Services Validated
*  Check whether Cloud Object Storage buckets with a quota have threshold-based alerts enabled







  NIST supplemental guidance 

Information handling and retention requirements cover the full life cycle of information, in some cases extending beyond the disposal of information systems. The National Archives and Records Administration provides guidance on records retention.
For more information, see the [enable.idempotence property](http://kafka.apache.org/documentation.htmlproducerconfigs).







 Throughput 

Throughput is expressed as the number of bytes per second that can be both sent and received in a cluster.



 Specific guidance for the Standard plan 

For throughput guidance information, see [Limits and quotas - Standard](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_quotasstandard_throughput).





 Specific guidance for the Enterprise plan 

For throughput guidance information, see [Limits and quotas - Enterprise](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_quotasenterprise_throughput).





 Specific guidance for the Satellite plan 

For throughput guidance information, see [Limits and quotas - Satellite](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_quotassatellite_throughput).





 Measurement 

Instrument applications to be aware of how they are performing. For example, the number of messages sent and received, message sizes, and return codes. Understanding an application's usage helps you configure its resources appropriately, such as the retention time for messages on topics.





 Saturation 

As the limit of the traffic that can be produced in to the cluster is approached, producers start to be throttled, latency increases, and ultimately errors such as timeout errors occur. Depending on the configuration, message consistency and durability might also be impacted. For more information, see [Consistency and durability of messages](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-slamessage_consistency).







 Consistency and durability of messages 

Kafka achieves its availability and durability by replicating the messages it receives across other nodes in the cluster, which can then be used in case of failure.
Provisioning an instance 

Before you can monitor and manage log data with IBM Log Analysis, you must first provision an instance of the service in IBM Cloud.

To provision an IBM Log Analysis instance in a Public Cloud region, you must select the service plan that is associated with the instance, the location where logs are collected, and the plan that determines the retention period for your logs. You can choose from 7, 14, or 30-day retention periods.

Alternatively, IBM Log Analysis offers a Lite plan that you can use to view your logs as they pass through the system. You can view logs by using log tailing. You can also design filters to prepare for upgrading to a longer retention period plan. This plan has a 0-day retention period.

You must also decide if you want your instance to be accessible through both public and private endpoints, or only private endpoints.



 Provisioning an instance through the Observability dashboard 

To provision an instance from the Observability dashboard in the IBM Cloud, complete the following steps:



1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

After you log in, the IBM Cloud UI opens.
2. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/icons/icon_hamburger.svg) > Observability to access the Observability dashboard.
3. Click Logging, then click Options > Create.
4. Enter a name for the service instance.
5. Select the [location](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-regions) where you plan to provision the instance.
6. Select a resource group.

By default, the default resource group is set.

Note: If you are not able to select a resource group, check that you have editing permissions on the resource group where you want to provision the instance.
7.
* All traffic is routed to the IBM Cloud private network.



monitoring agent data is collected at 10-seconds frequency. Data that is published by platform metrics is collected on a 1-minute frequency.



 Captures 

In IBM Cloud Monitoring, you can also enable captures when you configure a monitoring agent. A capture is a trace file that you can use to analyze what happens in a host during a time frame.



* Captures contain system calls, and other OS events.
* You can enable or disable this feature per node when you configure the monitoring agent that collects metrics from that node. A node can be a host, a container, a virtual machine, a bare metal, or any metrics source where you install a monitoring agent.



By default, the IBM instructions for configuring the monitoring agents disables the capture feature. If you choose to follow other instructions, IBM recommends disabling capture by setting the sysdig_capture_enabled: false in your dragent.yaml for Linux installations or the Kubernetes sysdig-agent Deployment custom resource.

When Captures are enabled, to avoid a security incident and potentially exposing data outside of your organization, check your organization's security policies before you enable captures on a node. Consider disabling the Capture feature for all your monitoring agents.







 How your data is stored in IBM Cloud Monitoring 

IBM Cloud Monitoring collects and aggregates metrics.



 Data location 

Metric data is hosted on the IBM Cloud.



* Each multi-zone region (MZR) location collects and aggregates metrics for each instance of the IBM Cloud Monitoring that runs in that location.
* Data is colocated in the region where the IBM Cloud Monitoring instance is provisioned. For example, metric data for an instance that is provisioned in US South is hosted in the US South region.







 Data retention 

Data is retained for each instance based on a roll-up policy.

As time progresses, the data is rolled up from a fine granularity to a coarser one by the end 2 months.

The roll-up policy describes the granularity of the data over time:



* Data is retained at 10-second resolution for the first 4 hours.
tolerations:
- key: "logging-monitoring"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 600
storage:
storageClassName: "ibmc-vpc-block-general-purpose"
size: 200G
resources:
requests:
memory: "24Gi"
proxy:
resources:
limits:
memory: 256Mi
requests:
memory: 256Mi
redundancyPolicy: "MultipleRedundancy"
visualization:
type: "kibana"
kibana:
replicas: 1
tolerations:
- key: "logging-monitoring"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 600
curation:
type: "curator"
curator:
schedule: "30 3 * * "
collection:
logs:
type: "fluentd"
fluentd: {}
Show more



The redundancyPolicy is set to MultipleRedundancy, which causes Elasticsearch to replicate shards to half of the data nodes (in this case, 2). To replicate shards to all three worker nodes, set redundancyPolicy to FullRedundancy.



You might need to adjust the following items:



* Application retention policy
* Infrastructure retention policy
* Audit retention policy
* The size of the block storage created for each Elasticsearch server (200 GB by default) to hold the logs expected to accumulate over the retention period



Verify that the logging instance is created by completing the following tasks.



1. Switch to the Workloads → Pods page.
2. Select the openshift-logging project. A list of several pods displays. The list of pods looks similar to the following list and includes pods for cluster logging, Elasticsearch, Fluentd, and Kibana:



* cluster-logging-operator-cb795f8dc-xkckc
As part of that action all namespaces, and the images that they contain, are removed in accordance with the data retention policy.

The IBM Cloud Container Registry data retention policy describes how long your data is stored after you delete the service. The data retention policy is included in the IBM Cloud Container Registry service description, which you can find in the Service Description for IBM Cloud in the [IBM Cloud Terms and Notices](https://cloud.ibm.com/docs/overview?topic=overview-terms).





 Deleting namespaces 

If you no longer require a registry namespace, you can remove the namespace from your IBM Cloud account. Deleting a namespace removes all images, trash, and trust information that is contained in the namespace. For more information, see [Removing namespaces](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_setup_cli_namespaceregistry_remove).





 Deleting images 

You can delete unwanted images from your private repository by using either the IBM Cloud console or the CLI. For more information, see [Deleting images from your private repository](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_images_registry_images_remove).

You can clean up your namespace by choosing to retain only the most recent images in each repository in that namespace in IBM Cloud Container Registry. You can detect and delete old images from all the repositories in a namespace by running a one-off command, ibmcloud cr retention-run, or by setting a scheduled policy by running the ibmcloud cr retention-policy-set command. For more information, see [Cleaning up your namespaces](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_retention).





 Deleting private repositories 

You can delete private repositories that are no longer required, and any associated images, by using the IBM Cloud console.
storageClassName: ibmc-file-silver
2. Create the PVC in your cluster.

kubectl apply -f pvc.yaml
3. After your silver-pvc PVC is bound, create an app deployment that uses your PVC. Create a file for your deployment and name it deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
labels:
app:
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- image: Your contanerized app image.
name: my-container
volumeMounts:
- name: my-volume
mountPath: /mount-path
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: silver-pvc
Show more
4. Create the deployment in your cluster.

kubectl apply -f deployment.yaml



For more information, see the following links.



* [Adding File Storage for Classic to apps](https://cloud.ibm.com/docs/containers?topic=containers-file_storageadd_file).
* [Storage class reference](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_storageclass_reference).
* [Custom storage classes](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_custom_storageclass).







 Deciding on the File Storage for Classic configuration 

IBM Cloud® Kubernetes Service provides pre-defined storage classes for File Storage for Classic that you can use to provision File Storage for Classic with a specific configuration.

Every storage class specifies the type of File Storage for Classic that you provision, including available size, IOPS, file system, and the retention policy.

After you provision a specific type of storage by using a storage class, you can't change the type, or retention policy for the storage device.
Additional fields for bucket-key-state.update events"} 







 Object events 

The following table lists the COS object events:



Table 4. Object events

 Action Description 

 cloud-object-storage.object-cors.read Get the CORS configuration 
 cloud-object-storage.object-acl.read Get the object ACL 
 cloud-object-storage.object-acl.create Create the object ACL 
 cloud-object-storage.object-retention-legal-hold.list List the legal holds on the object 
 cloud-object-storage.object-retention-legal-hold.update Add or remove object legal hold 
 cloud-object-storage.object-retention.update Extend the retention time 
 cloud-object-storage.object-expire.read Get when the object will expire 









 Data Events 

Data events are classified in the following categories:



* Bucket access events
* Object access events
* Multipart events
* Bucket versioning events





 Bucket access events 

The following table lists the COS bucket access events:



Table 5. Bucket access events

 Action Description 

 cloud-object-storage.bucket.list List the objects in the bucket 
 cloud-object-storage.bucket-metadata.read Get the metadata for the bucket 







 Object access events 

The following table lists the COS object access events:



Table 6. Object access events

 Action Description 

 cloud-object-storage.object-metadata.read Get the metadata for the object 
 cloud-object-storage.object.read Read the object 
 cloud-object-storage.object.create Create the object 
 cloud-object-storage.object.delete Delete the object 
 cloud-object-storage.objects.delete Delete multiple objects 
 cloud-object-storage.object-batch.delete Delete an object in a batch 
 cloud-object-storage.object-copy.read Read the source object to copy 
 cloud-object-storage.object-copy.create Create the target object from the copy 
 cloud-object-storage.object-restore.read Read the source object to restore 
 cloud-object-storage.object-restore.create Create the target object from the restore
Unlike the CREATE TABLE command, which does not have any existing rows, existing visible rows in the table are treated as if they were inserted by this ALTER TABLE transaction. The existing visible rows receive virtual insert timestamps that are equal to the retention start time. With these timestamps, the rows are potentially visible to time travel queries.





 Altering temporal schemas to nontemporal with the command-line 

To alter a temporal schema to nontemporal, set DATA_VERSION_RETENTION_TIME to 0.

For detailed syntax, the necessary privileges, and the CASCADE option, see [the ALTER SCHEMA command](https://www.ibm.com/docs/en/netezza?topic=npsscr-alter-schema-2).

ALTER SCHEMA <schema_name> DATA_VERSION_RETENTION_TIME 0 NOCASCADE;

Example:

ALTER SCHEMA SCHEMA DATA_VERSION_RETENTION_TIME 0 NOCASCADE;





 Altering nontemporal schemas to temporal with the command-line 

To alter a nontemporal schema to temporal, set DATA_VERSION_RETENTION_TIME to a nonzero value.

For detailed syntax, the necessary privileges, and the CASCADE option, see [the ALTER SCHEMA command](https://www.ibm.com/docs/en/netezza?topic=npsscr-alter-schema-2).

ALTER SCHEMA <schema_name> DATA_VERSION_RETENTION_TIME <number-of-days> NOCASCADE;

Example:

ALTER SCHEMA DB1 DATA_VERSION_RETENTION_TIME 30 NOCASCADE;





 Altering temporal databases to nontemporal with the command-line 

To alter a temporal database to nontemporal, set DATA_VERSION_RETENTION_TIME to 0.

For detailed syntax and the necessary privileges, see [the ALTER DATABASE command](https://www.ibm.com/docs/en/netezza?topic=npsscr-alter-database-2).

ALTER DATABASE <db_name> DATA_VERSION_RETENTION_TIME 0 NOCASCADE;

Example:

ALTER DATABASE DB1 DATA_VERSION_RETENTION_TIME 0 NOCASCADE;
* The default character_insertion_bias remains 0.0, but the models have changed in a manner that makes use of the parameter for speech recognition less necessary. For more information, see [Character insertion bias](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-parsinginsertion-bias).



Use the following guidelines when working with these parameters:



* If you use the default values for these parameters, continue to do so. The default values will likely continue to offer the best results for speech recognition.
* If you specify non-default values for these parameters, experiment with the default values. Your custom model might work well for speech recognition with the default values.
* If you feel that using different values for these parameters might improve speech recognition with your custom model, experiment with incremental changes to determine whether the parameters are needed to improve speech recognition.









 Acoustic model customization 

Acoustic model customization is available only for previous-generation models. It is not available for next-generation models.

The service was developed with base acoustic models that work well for various audio characteristics. But in cases like the following, adapting a base model to suit your audio can improve speech recognition:



* Your acoustic channel environment is unique. For example, the environment is noisy, microphone quality or positioning are suboptimal, or the audio suffers from far-field effects.
* Your speakers' speech patterns are atypical. For example, a speaker talks abnormally fast or the audio includes casual conversations.
* Your speakers' accents are pronounced. For example, the audio includes speakers who are talking in a non-native or second language.



The acoustic model customization interface can adapt a base model to your environment and speakers. You create a custom acoustic model and add audio data (audio resources) that closely match the acoustic signature of the audio that you want to transcribe. Once you train the custom acoustic model with your audio resources, you can use it for customized speech recognition.

The length of time that it takes the service to train the custom model depends on how much audio data the model contains. In general, training takes twice the length of the cumulative audio.
Using the Logpull service 

IBM customers can access the Logpull service on Enterprise accounts. This service allows users to consume request logs over HTTP using the [Logpull command](https://cloud.ibm.com/docs/cis?topic=cis-cli-plugin-cis-clilogpull-section). These logs contain data related to the connecting client, the request path through the network, and the response from the origin web server.



 Enabling log retention 

Edge logs are not retained by default. Before you can pull logs using the Logpull CLI, you must enable log retention. To do so, you must check the current setting, then turn log retention on or off.



1. To check whether log retention is currently turned off, use the log-retention CLI:



ibmcloud cis log-retention DNS_DOMAIN_ID



1. If the output shows that the flag is off (default), update the setting as follows.



ibmcloud cis log-retention-update DNS_DOMAIN_ID --flag on





 Logpull use cases 



 Getting logs based on RayID 

If you receive an error message after running a command, you can use the RayID provided in the response header to get the logs related to the command.

If you have a RAY_ID with -XXX on the end, be sure to remove it. For example, 12ab34cdef567gh8-XXX becomes 12ab34cdef567gh8.

Use the following command for the request:

ibmcloud cis logpull DNS_DOMAIN_ID --ray-id RAY_ID

The response follows:

{
"ClientIP": "68.278.11.89",
"ClientRequestHost": "testing.logpull.com",
"ClientRequestMethod": "GET",
"ClientRequestURI": "/var/www",
"EdgeEndTimestamp": 1545155129703000000,
"EdgeResponseBytes": 1935,
"EdgeResponseStatus": 403,
It is expected - proceed by sending the form.







 Configuring a BMR backup job 



1. In the navigation, click Computers, then the expansion arrow to display the information of the selected server.
2. Click Configure Manually. The vault settings page is loaded.
3. Click Add Vault.
4. Expand the Vault Profile menu and select the vault. All values auto-populate.
5. Click Save.
6. Click the Jobs tab.
7. From the Select Job Task menu, select Create Bare Metal Restore Job.
8. In the Create New Job window, enter a Job Name and a Job Description.
9. Select the files and folders that you want to include in the backup.
10. Enter the encryption password into the Password and Confirm Password fields. You can also add a Password Hint. You need this password to restore files from the backup. Without the password, you can't restore an encrypted backup. Lost password cannot be recovered in any way.
11. Click Apply now to confirm the backup sets.
12. You can leave the Advanced Backup Options with their default settings. If you want detailed log files for the backup job, you can enable them by expanding the Log Detail Level menu and selecting File.
13. Click Create Job.
14. IBM Cloud Backup for Classic offers three job retention schemes: Daily, Weekly, Monthly. In the new window, select the appropriate retention period and click Save.



For more information about Retention Schemes, see the [FAQ](https://cloud.ibm.com/docs/Backup?topic=Backup-faqsfaqs).





 Running a BMR backup job 



* If you scheduled a time-based backup job, you don't need to do anything else. Your job runs automatically as scheduled.
* If you set up a manual job (without a time-based schedule), you can run it by selecting its row in the job list and click Run backup. As with time-based jobs, you can choose the Retention Scheme and the Advanced backup options.