vania-janet/multiturn-rag-retrieval-data huggingface dataset:
FAQ - Access Control and Encryption 

Frequently asked questions can produce helpful answers and insight into best practices for working with IBM Cloud® Object Storage.



 What types of authentication can I use to access IBM Cloud® Object Storage? 

You can use an OAuth 2 token or an HMAC key for authentication. The HMAC key can be used for S3-compatible tools such as rclone, Cyberduck, and others.



* For instructions to obtain an OAuth token, see [Generating an IBM Cloud IAM token by using an API key](https://cloud.ibm.com/docs/account?topic=account-iamtoken_from_apikey).
* For instructions to obtain the HMAC credentials, see [Using HMAC Credentials](https://cloud.ibm.com/docs/cloud-object-storage/hmac?topic=cloud-object-storage-uhc-hmac-credentials-main).



Also, see [API Key vs HMAC](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-service-credentialsservice-credentials-iam-hmac).





 How do cross-origin resource sharing (CORS) and a bucket firewall differ in limiting access to data? 

CORS allows interactions between resources from different origins that are normally prohibited. A bucket firewall allows access only to requests from a list of allowed IP addresses. For more information on CORS, see [What is CORS?](https://cloud.ibm.com/docs/CDN?topic=CDN-cors-and-cors-requests-through-your-cdnwhat-is-cors).





 How do I allow Aspera High-Speed Transfer through a bucket with context-based restrictions or a firewall? 

The full list (in JSON) of Aspera High-Speed Transfer IP addresses that are used with IBM Cloud Object Storage can be found [using this API endpoint](https://ats.aspera.io/pub/v1/servers/softlayer).





 Does Object Storage provide encryption at rest and in motion? 

Yes.
Migrating from AWS 

Migrating your data is a complex and daunting task, but don't let it stop you from making the right decision. Using the right tools will make moving from Amazon™ to IBM Cloud® Object Storage more secure, globally accessible, and completed with confidence.



 Before you begin 

Determine your goals and process for your migration before starting your migration. You may also consider training and partnerships to be beneficial. Your planning and assessment stage will consider many possibilities, including security and technical capabilities.

Documentation for any project will help keep you keep track of your resources as well as your goals. After assessing your existing projects, you may benefit by updating them to use IBM Cloud Object Storage libraries like those for ([Java](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-java), [Python](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-python), [Node.js](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-node)). If you're interested in programmer interfaces, the [REST API](https://cloud.ibm.com/docs/cloud-object-storage/api-reference?topic=cloud-object-storage-compatibility-api) will provide an in-depth look at operations and configurations.





 Provision and configure IBM Cloud Object Storage 



1. If you haven't already, create an instance of IBM Cloud Object Storage from the [Console](https://cloud.ibm.com/catalog/services/cloud-object-storage).
2. Create any buckets that you anticipate will be needed to store your transferred data.
IBM Cloud Classic Block Storage [IBM Cloud Classic Block Storage](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-startedgetting-started) is persistent, high-performance iSCSI storage that you can add to your apps by using Kubernetes persistent volumes (PVs). Use block storage to deploy stateful apps in a single zone or as high-performance storage for single pods. For more information about how to provision block storage in your cluster, see [Setting up IBM Cloud Block Storage](https://cloud.ibm.com/docs/containers?topic=containers-block_storageblock_storage) Yes 
 Block Storage for VPC [Block Storage for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-block-storage) provides hypervisor-mounted, high-performance data storage for your virtual server instances that you provision within a VPC cluster. For more information about how to provision VPC Block Storage in your cluster, see [Setting up Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-block) Yes 
 IBM Cloud Object Storage Data that is stored with Object Storage is encrypted and dispersed across multiple geographic locations, and accessed over HTTP by using a REST API. You can use the [ibm-backup-restore image](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesibmcloud-backup-restore) to configure the service to make one-time or scheduled backups for data in your clusters. For more information about the service, see the [Object Storage documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage). Yes Yes
* [Non-VRF or VRF-enabled account with public cloud service endpoint only](https://cloud.ibm.com/docs/containers?topic=containers-service-archno-vrf-public-endpoint)
* [VRF-enabled account with private and public cloud service endpoints](https://cloud.ibm.com/docs/containers?topic=containers-service-archvrf-both-endpoints)
* [Kubernetes master components](https://cloud.ibm.com/docs/containers?topic=containers-service-archmaster-components)
* [Worker node components](https://cloud.ibm.com/docs/containers?topic=containers-service-archworker-components)
* [kube-system namespace](https://cloud.ibm.com/docs/containers?topic=containers-service-archkube-system-ns-resources)
* [ibm-system namespace](https://cloud.ibm.com/docs/containers?topic=containers-service-archibm-system-ns-resources)
* [default namespace](https://cloud.ibm.com/docs/containers?topic=containers-service-archdefault-ns-resources)



* [VPC cluster](https://cloud.ibm.com/docs/containers?topic=containers-service-archarchitecture_vpc)
* [Overview of personal and sensitive data storage and removal options](https://cloud.ibm.com/docs/containers?topic=containers-service-archibm-data)



* [What information is stored with IBM when using IBM Cloud Kubernetes Service?](https://cloud.ibm.com/docs/containers?topic=containers-service-archpi-info)
* [How is my information stored and encrypted?](https://cloud.ibm.com/docs/containers?topic=containers-service-archpi-storage)
* [Where is my information stored?]
[Cloud Object Storage bucket toggle](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/cos-evidence.png)

Figure 12. COS bucket toggle

Cloud Object Storage is used to store the evidence and artifacts that are generated by the Compliance Pipelines. If you want to use this feature, you must have a Cloud Object Storage instance and a Bucket. For more information, follow the steps [here](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cos-config).

You can set any type of Cloud Object Storage bucket as a locker, even without a retention policy. The pipeline does not currently check or enforce settings.

For help, see the [Cloud Object Storage documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage).

You need to provide the following information for the Pipelines to reach the bucket:



* Cloud Object Storage endpoint
* Bucket name
* Service API key



You can set up the Cloud Object Storage locker later, by providing the necessary cos-bucket-name and cos-endpoint.

To get the Cloud Object Storage Endpoint, go to your Cloud Object Storage Instance's page and select the Endpoints section in the menu. Copy the public, priavte, or direct endpoint that matches the bucket's region and resiliency.

Zoom

![DevSecOps Cloud Object Storage Endpoint menu](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-cos-endpoint-menu.png)

Figure 13. DevSecOps Cloud Object Storage Endpoint menu
Comparing IBM Cloud Object Storage to FTP 

The File Transfer Protocol, (FTP) is a popular way to transfer files, but how does it compare to IBM Cloud® Object Storage?

IBM Cloud Object Storage stores encrypted and dispersed data across multiple geographic locations. The information is accessible over popular protocols like HTTPS using a modern RESTful API. FTP, by contrast, requires both a client and a server application and uses an insecure protocol by default. Object Storage does all the work of the server FTP daemon (ftpd) and offers more options for security and validation than can be obtained from ftpd or similar services.



 How IBM Cloud Object Storage is similar to FTP 

If you've used FTP in the past, you have either worked from the command line or from a client application that uses a GUI. Cyberduck is a popular, open source, and easy-to-use graphical interface for either IBM Cloud Object Storage or FTP.

Cyberduck provides full operational visibility in connecting to IBM Cloud Object Storage. Cyberduck is downloaded from [cyberduck.io/](https://cyberduck.io/). Once you have it installed, you can configure it to connect to your instance of Object Storage.

Use Cyberduck to create a connection to IBM Cloud Object Storage. Then, synchronize a folder of local files to a bucket. After you complete [getting started](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage) Object Storage and obtained your [credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials), follow these steps:



1. Download, install, and start Cyberduck.
2. When the application opens, you can create a connection to Object Storage. Click Open Connection to configure the connection.
3. A pop-up window opens. From the menu, select the option, Amazon S3. Enter your information into the following fields:



* Server: enter the appropriate endpoint for your data at IBM Cloud Object Storage
[Setting up classic IBM Cloud File Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_storage)



* [Quick start for IBM Cloud Classic File Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_qs)
* [Deciding on the File Storage for Classic configuration](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_predefined_storageclass)



* [File storage types](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile-types)
* [IOPS](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile-iops)
* [Reclaim policy](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile-reclaim)
* [Billing type](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile-billing)



* [Adding File Storage for Classic to apps](https://cloud.ibm.com/docs/containers?topic=containers-file_storageadd_file)
* [Using existing File Storage for Classic in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-file_storageexisting_file)



* [Preparing your existing storage](https://cloud.ibm.com/docs/containers?topic=containers-file_storageexisting-file-1)
* [Creating a persistent volume and a persistent volume claim](https://cloud.ibm.com/docs/containers?topic=containers-file_storageexisting-file-2)



* [Using File Storage for Classic in a stateful set](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_statefulset)
* [Backing up and restoring PVC data for file and block storage](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesibmcloud-backup-restore)



* [Setting up an IBM Cloud Object Storage service instance](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesbackup_restore_setup_object_storage)
* [Using IBM Cloud Object Storage to back up and restore PVC data](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesbackup-restore-pvc)



* [Setting up IBM Cloud Monitoring for storage volumes](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmonitor_storage)



* [Troubleshooting persistent storage when an IBM Cloud Monitoring alert is triggered](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmonitor_storage_ts)











 Enhancing cluster capabilities with integrations 

[IBM Cloud Kubernetes Service partners](https://cloud.ibm.com/docs/containers?topic=containers-service-partnersservice-partners)



* [Portworx](https://cloud.ibm.com/docs/containers?topic=containers-service-partnersportworx-parter)



* [Benefits](https://cloud.ibm.com/docs/containers?topic=containers-service-partnersportworx-benefits)
* [Integration with IBM Cloud Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-service-partnersportworx-integration)
* [Billing and support](https://cloud.ibm.com/docs/containers?topic=containers-service-partnersportworx-billing-support)





[IBM Cloud services and third-party integrations](https://cloud.ibm.com/docs/containers?topic=containers-ibm-3rd-party-integrationsibm-3rd-party-integrations)
"backup", // the name of the Bucket to which the Object will be copied,
"myFile.txt" // and the name of the copied Object in the destination Bucket
);

SDK References



* Classes



* [ObjectMetadata](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/ObjectMetadata.html)
* [PutObjectRequest](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/PutObjectRequest.html)



* Methods



* [putObject](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.htmlputObject-com.ibm.cloud.objectstorage.services.s3.model.PutObjectRequest-)







 putObject Exception 

The putObject method might throw the following exception even if the new object upload was successful:

Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1597)
at ibmcos.CoSExample.createTextFile(CoSExample.java:174)
at ibmcos.CoSExample.main(CoSExample.java:65)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190)
Using cURL 

You can get the most out working with the command line in most environments with IBM Cloud® Object Storage and cURL.

Here's a 'cheat sheet' of basic curl commands for the IBM Cloud® Object Storage REST API. More detail can be found in the API reference for [buckets](https://cloud.ibm.com/docs/cloud-object-storage/api-reference?topic=cloud-object-storage-compatibility-api-bucket-operations) or [objects](https://cloud.ibm.com/docs/cloud-object-storage/api-reference?topic=cloud-object-storage-object-operations).

Using curl assumes a certain amount of familiarity with the command line and Object Storage, and have the necessary information from a [service credential](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-service-credentials), the [endpoints reference](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints), or the [console](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage). If any terms or variables are unfamiliar, they can be found in the [glossary](https://cloud.ibm.com/docs/overview?topic=overview-glossary).

Note: Personally Identifiable Information (PII): When naming buckets or objects, do not use any information that can identify any user (natural person) by name, location, or any other means.



 Request an IAM Token 

Two ways you can generate an IAM oauth token for authenticating requests are using a curl command with an API key (described later), or from the command line by using [IBM Cloud® CLI](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_commands_iamibmcloud_iam_oauth_tokens).



 Request an IAM token by using an API key 

Ensure that you have an API key.
(https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-ports)
* [How can I estimate the IBM Blockchain Platform sizing requirements for my development, test, and production environments?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-sizing)
* [How does pricing work on the IBM Blockchain Platform for IBM Cloud?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cloud-pricing)
* [What are the limitations of the free IBM Blockchain Platform using the IBM Cloud Kubernetes Service free cluster?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-free-cluster)
* [What regions or locations are available for the IBM Blockchain Platform for IBM Cloud?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-IBP-Overview-1-9)
* [What persistent file storage does IBM Blockchain Platform for IBM Cloud use by default?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cloud-storage)
* [Do I need multizone region storage for IBM Blockchain Platform nodes?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cloud-mzr-storage)
* [What versions of Red Hat OpenShift are supported?]
The tasks a bot is programmed to do are fairly simple, but the bot can do these tasks repeatedly at a much faster rate than a human can.

Bots don't access the internet using a traditional web browser or a mouse to interact with visual content. Bots are software programs that typically use a "headless browser" to make HTTP requests and other activities.

Bots can do almost any repetitive, non-creative task that can be automated, including filling out forms, reading and downloading content, and even hold basic conversations with humans as chatbots. Like any tool that can be used for good, bots can be used for malicious behavior, too.





 Differences between good bots and bad bots 

It is estimated that up to half of all internet traffic is bot traffic. Some bots are malicious, and some are "good."

Bots that misuse online products or services can be considered "bad." Bad bots range from malicious to simply annoying; for instance, breaking into user accounts to steal data or buying concert tickets online to assist scalpers.

A bot that performs a helpful service can be considered "good." For example, customer service chatbots, search engine crawlers, and performance monitoring bots are generally good bots. Good bots look for and abide by the rules outlined in a website's robots.txt file.





 The robots.txt file 

Robots.txt is a file that outlines the rules for bots accessing properties on a web server, though the file itself does not enforce these rules. Anyone programming a bot is expected to follow an honor system and make sure that their bot checks a website's robots.txt file before accessing the website. Malicious bots don't follow this system, which generates the need for bot management.





 How bot management works 

To identify bots, bot managers might use JavaScript challenges (which determine whether a traditional web browser is being used) or CAPTCHA challenges. They can also determine which users are humans and which are bots by behavioral analysis; by comparing a user's behavior to the standard behavior of users in the past.

When a bot is identified as bad, it can be redirected to a different page or blocked from accessing a web resource altogether.
How bot management works 

To identify bots, bot managers might use JavaScript challenges (which determine whether a traditional web browser is being used) or CAPTCHA challenges. They can also determine which users are humans and which are bots by behavioral analysis; by comparing a user's behavior to the standard behavior of users in the past.

When a bot is identified as bad, it can be redirected to a different page or blocked from accessing a web resource altogether.

Good bots can be added to an allowlist. A bot manager can also distinguish between good and bad bots by using further behavioral analysis.

Another bot management approach is to use the robots.txt file to set up a honeypot. A honeypot is a fake target for bad actors that, when accessed, exposes the bad actor as malicious. In the case of a bot, a honeypot could be a webpage on the site that's forbidden to bots by the robots.txt file. Good bots will read the robots.txt file and avoid that webpage; some bad bots will crawl the webpage. By tracking the IP address of the bots that access the honeypot, bad bots can be identified and blocked.





 Kinds of bot attacks bot management mitigates 

A bot management solution can help stop a variety of attacks, including the following:



* DDoS attacks
* DoS attacks
* Credential stuffing
* Credit card stuffing
* Brute force password cracking
* Spam content
* Data scraping and web scraping
* Email address harvesting
* Ad fraud
* Click fraud



These bot activities are not always considered "malicious," but a bot manager should still be able to mitigate them:



* Inventory hoarding
* Automated posting on social forums or platforms
* Shopping cart stuffing







 How does CIS manage bots? 

CIS collects data from requests flowing through its network every day. With this data, powered by Cloudflare's machine learning and behavior analysis, CIS can identify likely bot activity and can provide you with information on how to allow or disallow specific bot traffic using Firewall Rules.

Cloudflare Bot Management uses the following detection mechanisms, each producing their own scores, which are then combined to form a single score:
About bot management 

Bot management blocks undesired or malicious internet bot traffic while allowing useful bots to access web properties. Bot management detects bot activity, discerns between desirable and undesirable bot behavior, and identifies the sources of the undesirable activity.

Unmanaged bots can cause serious issues for web properties. Excessive bot traffic can put a heavy load on web servers, slowing or denying service to legitimate users (for example, DDoS attacks). Malicious bots can perform cyber attacks such as scraping content from websites, stealing user credentials, and spreading spam content.



 What bot managers do 

A bot manager is any software product that manages bots. Bot managers should be able to block some bots and allow others through, instead of simply blocking all non-human traffic. If all bots are blocked and Google bots aren't able to index a page, for instance, then that page can't show up in Google search results, resulting in greatly reduced organic traffic to the website.

An effective bot manager accomplishes the following goals:



* Distinguishes bots from human visitors
* Identifies bot reputation
* Identifies bot origin IP addresses and block based on IP reputation
* Analyzes bot behavior
* Adds "good" bots to allowlists
* Challenges potential bots with a CAPTCHA test, JavaScript injection, or other methods
* Rate limits any potential bot that is over-using a service
* Denies access to certain content or resources for "bad" bots
* Serves alternative content to bots







 What are bots and what do they do? 

A bot is a computer program that automatically does certain actions on a network. The tasks a bot is programmed to do are fairly simple, but the bot can do these tasks repeatedly at a much faster rate than a human can.

Bots don't access the internet using a traditional web browser or a mouse to interact with visual content. Bots are software programs that typically use a "headless browser" to make HTTP requests and other activities.

Bots can do almost any repetitive, non-creative task that can be automated, including filling out forms, reading and downloading content, and even hold basic conversations with humans as chatbots.