vania-janet/multiturn-rag-retrieval-data huggingface dataset:
1. Create a service ID and set it as an environment variable.

export SERVICE_ID=ibmcloud iam service-id-create secure-file-storage-tutorial --description "A service ID for e2e security tutorial." --output json | jq -r ".id"; echo $SERVICE_ID
2. Assign the service ID permissions to read secrets from Secrets Manager.

ibmcloud iam service-policy-create $SERVICE_ID --roles "SecretsReader" --service-name secrets-manager
3. Create an API key for your service ID.

export IBM_CLOUD_API_KEY=ibmcloud iam service-api-key-create secure-file-storage-tutorial $SERVICE_ID --description "An API key for e2e security tutorial." --output json | jq -r ".apikey"
4. Create a secret in your cluster for that API key.

kubectl -n default create secret generic secure-file-storage-api-key --from-literal=apikey=$IBM_CLOUD_API_KEY
5. Run the following commands to install the External Secrets Operator.

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets external-secrets/external-secrets
6. Verify the values for MYDOMAIN, SECRETS_MANAGER_API_URL and PUBLIC_CERT_ID have been exported into the environment:

echo MYDOMAIN $(printenv MYDOMAIN)
echo SECRETS_MANAGER_API_URL $(printenv SECRETS_MANAGER_API_URL)
echo PUBLIC_CERT_ID $(printenv PUBLIC_CERT_ID)

You can test the secrets manager variables with this command:

ibmcloud sm secret --service-url $SECRETS_MANAGER_API_URL --secret-type public_cert --id $PUBLIC_CERT_ID
7. Run the below command to generate a new copy of secure-file-storage.yaml.
If the intention that the credential be used to grant, access to a subset of buckets and not the entire instance, this policy needs to be edited. See the [Bucket permissions](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-iam-bucket-permissions) page for more details.





 API Key vs HMAC 

In general IAM API Keys are the preferred method of authentication for IBM Cloud® Object Storage. HMAC is supported primarily for compatibility with an earlier version with applications which migrated from IaaS {site.data.keyword.cos_short}} and legacy S3 applications. IAM is also natively supported when developing applications with the COS SDKs. Token expiration and refresh are handled automatically to simplify the process.

For more information about IAM visit - [Getting started with IAM](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam)

For more information about HMAC visit - [Using HMAC Credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main)
Click Create.









 Step 6: Secure with HTTPS 

HTTPS encryption requires signed certificates to be accessible from both the CIS GLB and the VPC load balancers. Below the IBM Cloud Secrets Manager will be used to order or import and then manage the certificate for your domain. Identity and Access Management (IAM) service authorization is then configured to allow read access to the certificate from the desired service.



 Create and authorize a Secrets Manager instance 



1. If you have an existing [Secrets Manager](https://cloud.ibm.com/catalog/services/secrets-manager) instance, you can use it for this tutorial or create a new one if needed by following the steps outlined in [Creating a Secrets Manager service instance](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-create-instance&interface=ui).
2. Create an authorization that gives the VPC load balancer service access to the Secrets Manager instance that contains the SSL certificate.



* Navigate to [Identity and Access Authorizations](https://cloud.ibm.com/iam/authorizations).
* Click Create and select VPC Infrastructure Services as the source service.
* Load Balancer for VPC as the resource type.
* Secrets Manager as the target service.
* Assign the Writer service access role.
* To create a load balancer, you must grant All resource instances authorization for the source resource instance. The target service instance may be All instances, or it may be your specific Secrets Manager instance.
* Click on Authorize.



3. Continuing in the Authorizations panel, create an authorization that gives the Secrets Manager access to CIS:



* Click Create and choose Secrets Manager as the source service.
* Choose All instances or just the Secrets Manager created earlier.
* Internet Services as the target service.
* Choose All instances or just the CIS created earlier.
* Assign the Manager service access role.
* Click on Authorize.



If your CIS instance supports multiple domains, you can also assign Reader role to the CIS instance and Manager to the specific domain that is you are using for your solution.
How IBM Cloud IAM works 

Learn about what IBM Cloud Identity and Access Management (IAM) is, how IAM works, what features are available, and how to access the console, CLI, and APIs to work with IAM in your account.

IAM enables you to securely authenticate users for platform services and control access to resources consistently across IBM Cloud. A set of IBM Cloud services is enabled to use IBM Cloud IAM for access control, and are organized into [resource groups](https://cloud.ibm.com/docs/account?topic=account-rgs) within your account so you can give users access quickly to more than one resource at a time. Each of these services is labeled as "IAM-enabled" in the catalog. You can use IAM access policies to assign users, service IDs, and trusted profiles access to resources within your account. And, you can group users, service IDs, and trusted profiles into an [access group](https://cloud.ibm.com/docs/account?topic=account-groups) to easily give all members of the group the same level of access.

You can use [trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile) to automate the grouping and granting of access to users, services, and app identities. By specifying conditions based on SAML attributes for users whose identity is federated from your external identity provider (IdP), users can be granted access to resources without having to be invited to the account if they meet those conditions. For service and app identities, you can define fine-grained authorization for all applications that are running in a compute resource without creating service IDs or managing the API key lifecycle for applications.

Zoom

![IAM access control in an account](https://cloud.ibm.com/docs-content/v1/content/d4595e5202a9a27767cf034e81b038cdf772e0d5/secure-enterprise/includes/account/includes/account/images/tp-in-ag-access-diagram.svg)

Figure 1.
ibmcloud secrets-manager config-element-update --secret-type=public_cert --config-element=certificate_authorities --config-name=exampleString --type=cis --config='{"anyKey": "anyValue"}'







 ibmcloud secrets-manager config-element-action 

Invoke an action on a specified configuration element. This method supports the following actions:



* sign_intermediate: Sign an intermediate certificate authority.
* sign_csr: Sign a certificate signing request.
* set_signed: Set a signed intermediate certificate authority.
* revoke: Revoke an internally signed intermediate certificate authority certificate.
* rotate_crl: Rotate the certificate revocation list (CRL) of an intermediate certificate authority.



ibmcloud secrets-manager config-element-action --secret-type SECRET-TYPE --config-element CONFIG-ELEMENT --config-name CONFIG-NAME --action ACTION [--config CONFIG]



 Command options 

--secret-type (string)
: The secret type. Required.

Allowable values are: private_cert.

--config-element (string)
: The configuration element on which the action is applied. Required.

Allowable values are: root_certificate_authorities, intermediate_certificate_authorities.

--config-name (string)
: The name of the certificate authority. Required.

--action (string)
: The action to perform on the specified configuration element. Required.

Allowable values are: sign_intermediate, sign_csr, set_signed, revoke, rotate_crl.

--config ([ConfigAction](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-secrets-manager-cli-v1cli-config-action-example-schema-v1))
: Properties that describe an action on a configuration element.





 Examples
From the list of instances, select the Event Notifications service instance that you want to authorize.
4. Select the Event Source Manager role.
5. Click Review.
6. Click Assign.



7. To confirm the connection, click Connect.

A success message is displayed to indicate that Secrets Manager is now connected to Event Notifications. If you need to disconnect from Event Notifications later, you can use the options menu ![options icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Disconnect to remove the Secrets Manager as a source service in the Event Notifications instance.

If you choose to disconnect Event Notifications, do not delete the IAM authorization between Secrets Manager and Event Notifications. Secrets Manager uses the existing authorization to unregister from Event Notifications. If an Event Notifications instance is deleted, any authorizations that exist between the service and the Secrets Manager are also deleted by IAM.







 Connecting to Event Notifications with the API 

Before you can enable notifications for Secrets Manager, be sure that you have an [Event Notifications service instance](https://cloud.ibm.com/catalog/services/event-notifications) that is in the same account as your Secrets Manager instance. Then, you can connect to Event Notifications programmatically by calling the Secrets Manager API.

The following example shows a query that you can use to register your Secrets Manager source details with Event Notifications. When you call the API, replace the ID variables and IAM token with the values that are specific to your Secrets Manager instance.

You can find the event_notifications_instance_crn value in the console by going to the Resource list and clicking the Event Notifications instance row.

curl -X POST
--H "Authorization: Bearer {iam_token}" --H "Accept: application/json" --H "Content-Type: application/json" --d'{
* UI
* CLI
* API







 Protecting Security and Compliance Center resources with context-based restrictions 

Context-based restrictions give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud resources based on the context of access requests. Access to Security and Compliance Center resources can be controlled with context-based restrictions and identity and access management (IAM) policies.

These restrictions work with traditional IAM policies, which are based on identity, to provide an extra layer of protection. Unlike IAM policies, context-based restrictions don't assign access. Context-based restrictions check that an access request comes from an allowed context that you configure. Since both IAM access and context-based restrictions enforce access, context-based restrictions offer protection even in the face of compromised or mismanaged credentials. For more information, see [What are context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-whatis).

A user must have the Administrator role on the Security and Compliance Center service to create, update, or delete rules. A user must also have either the Editor or Administrator role on the Context-based restrictions service to create, update, or delete network zones. A user with the Viewer role on the Context-based restrictions service can add only network zones to a rule.

Any IBM Cloud Activity Tracker or audit log events generated come from the context-based restrictions service, not Security and Compliance Center. For more information, see [Monitoring context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-cbr-monitor).

To begin protecting your Security and Compliance Center resources with context-based restrictions, see the tutorial for [Leveraging context-based restrictions to secure your resources](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-tutorial).



 How Security and Compliance Center integrates with context-based restrictions 

Context-based restrictions that are set to Security and Compliance Center apply holistically to all components of Security and Compliance Center. By enabling context-based restrictions, you can restrict access to Security and Compliance Center’s user interface features and Security and Compliance Center’s public APIs.
IA-5 (2) - Pki-based Authentication 



  Control requirements 

The information system, for PKI-based authentication:

IA-5 (2) (a)
:   Validates certifications by constructing and verifying a certification path to an accepted trust anchor including checking certificate status information;

IA-5 (2) (b)
:   Enforces authorized access to the corresponding private key;

IA-5 (2) (c)
:   Maps the authenticated identity to the account of the individual or group; and

IA-5 (2) (d)
:   Implements a local cache of revocation data to support path discovery and validation in case of inability to access revocation information via the network.





  Additional IBM Cloud for Financial Services specifications 



*  TLS certificate revocation status checking, and revocation status caching is only applicable to connections between a browser and server.







  NIST supplemental guidance 

Status information for certification paths includes, for example, certificate revocation lists or certificate status protocol responses. For PIV cards, validation of certifications involves the construction and verification of a certification path to the Common Policy Root trust anchor including certificate policy processing.
This file is the design blueprint for your toolchain and includes all of the tool integrations that are available when you create a toolchain instance based on that template. To customize an existing toolchain template to include a Secrets Manager tool integration, insert a YAML definition.

sm-compliance-secrets:
service_id: secretsmanager
parameters:
name: sm-compliance-secrets
instance-id-type: instance-name
region: us-south
resource-group: default
instance-name: ffs-secrets







 Identifying secrets by CRN 

When you configure the Secrets Manager tool integration to identify secrets by CRN, your toolchain can access arbitrary, key-value, and IAM credential secrets that are stored in the Secrets Manager. For more information about the different types of secrets in Secrets Manager, see [Working with secrets of different types](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-what-is-secretsecret-types). You can use only entire key-value secrets for CRN secrets, not individual keys.

Currently, you cannot configure a Secrets Manager tool integration to identify secrets by CRN during toolchain creation. Instead, complete the configuration after you create the toolchain.

Configure Secrets Manager to securely manage secrets that are part of your toolchain:



1. If you have a toolchain and are adding this tool integration to it, from the IBM Cloud console, click the menu icon ![hamburger icon](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/icon_hamburger.svg) and select DevOps. On the Toolchains page, click the toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous Delivery card, click View toolchain. Then, click Overview.

a. Click Add tool.

b. In the Tool Integrations section, click Secrets Manager.
For example, the following command creates an intermediate CA with internal signing.

ibmcloud secrets-manager configuration-create
--configuration-prototype='{
"config_type": "private_cert_configuration_intermediate_ca",
"name": "example-intermediate-CA",
"max_ttl": "26300h",
"signing_method": "internal",
"issuer": "example-root-CA",
"crl_expiry": "72h",
"crl_disable": false,
"crl_distribution_points_encoded": true,
"issuing_certificates_urls_encoded": true,
"common_name": "example.com",
"alt_names": [
"alt-name-1","alt-name-2"
],
"ip_sans": "127.0.0.1",
"uri_sans": "https://www.example.com/test",
"other_sans": ["1.2.3.5.4.3.201.10.4.3;utf8:test@example.com"],
"format": "pem",
"private_key_format": "der",
"key_type": "rsa",
"key_bits": 4096,
"exclude_cn_from_sans": false
}'





 Step 2: Sign the intermediate CA 

To sign the intermediate CA from the Secrets Manager CLI, apply the sign-intermediate action by runnning the [ibmcloud secrets-manager configuration-action-create](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-cli-plugin-secrets-manager-clisecrets-manager-cli-configuration-action-create-command) command. Pass the name of the signing certificate authority (the issuer) with the --name command option.
"serial_number": "1:2:3:4"
},
],
"source_instance_api_private_url": "https://<instance_id>.private.<region>.secrets-manager.appdomain.cloud/api",
"source_instance_api_public_url": "https://<instance_id>.<region>.secrets-manager.appdomain.cloud/api",
"source_instance_crn": "crn:v1:bluemix:public:secrets-manager:<region>:a/<account_id>:<instance_id>::",
"source_instance_dashboard_url": "https://cloud.ibm.com/services/secrets-manager/crn%3Av1%3Abluemix%3Apublic%3Asecrets-manager%3A<region>%3Aa%2Fa6cc9f5f21f34323a4175c1117638b40%3A<instance_id>%3A%3A",
"source_service": "SecretsManager"
}

Review following table for more information about event notification properties.



Table 3. Properties in an event notification payload

 Property Description 

 event_sub_type The subtype that corresponds with the [type of event](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-event-notifications&interface=uievent-notifications-list) that triggered a notification. 
 event_type The [type of event](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-event-notifications&interface=uievent-notifications-list) that triggered a notification. 
 secret_type The type of secret that is associated with the event.<br><br>Possible values include: imported_cert, private_cert, public_cert 
 secrets[] A list of objects that contain the metadata properties of a secret that is associated with the event.
How IBM Cloud IAM works 

Learn about what IBM Cloud Identity and Access Management (IAM) is, how IAM works, what features are available, and how to access the console, CLI, and APIs to work with IAM in your account.

IAM enables you to securely authenticate users for platform services and control access to resources consistently across IBM Cloud. A set of IBM Cloud services is enabled to use IBM Cloud IAM for access control, and are organized into [resource groups](https://cloud.ibm.com/docs/account?topic=account-rgs) within your account so you can give users access quickly to more than one resource at a time. Each of these services is labeled as "IAM-enabled" in the catalog. You can use IAM access policies to assign users, service IDs, and trusted profiles access to resources within your account. And, you can group users, service IDs, and trusted profiles into an [access group](https://cloud.ibm.com/docs/account?topic=account-groups) to easily give all members of the group the same level of access.

You can use [trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile) to automate the grouping and granting of access to users, services, and app identities. By specifying conditions based on SAML attributes for users whose identity is federated from your external identity provider (IdP), users can be granted access to resources without having to be invited to the account if they meet those conditions. For service and app identities, you can define fine-grained authorization for all applications that are running in a compute resource without creating service IDs or managing the API key lifecycle for applications.

Zoom

![IAM access control in an account](https://cloud.ibm.com/docs-content/v1/content/814aeaf04800bb4f74ea1c057d4abb166a5f357a/account/images/tp-in-ag-access-diagram.svg)

Figure 1. How IAM access works in an account by using access groups
If you use IAM, ensure that the user ID that you use to submit the SQL query through the web console or the API is authorized to read the input locations (referenced in the FROM clauses of the query), and to write the result set to the target location (referenced in the INTO clause of the query). This mechanism provides seamless single sign-on (SSO) for your query submissions in Data Engine. You can use the following alternative methods for authenticating access to data resources:



 IBM Cloud service credentials 

You can use the unique CRN of a service instance in IBM Cloud to address a resource location. The credentials for accessing this data resource are retrieved from the Credentials object of that service instance. The IAM user ID that is used to submit the query must have the operator role for the service instance of the corresponding data resource.





 Custom user ID and password for each data resource 

You can use the USER and PASSWORD keywords in the corresponding FROM or INTO clauses to securely pass user ID and password for each resource location. To ensure that the passing of sensitive data is secure, store the password as an arbitrary secret in an instance of IBM Cloud® Secrets Manager to which you have access, and then use the CRN of this arbitrary secret instead of the plain text password. For more information, see [Storing arbitrary secrets](https://cloud.ibm.com/docs/%5B/docs/sql-query?topic=sql-query-kpsetup%5D%28https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-arbitrary-secrets&interface=ui%29) about how to store the password securely.

In addition, you also have the option to store the password as a custom standard key in an instance of IBM® Key Protect to which you have access, and then use the CRN of this custom standard key instead of the plain text password. For more information, see [Setting up custom secrets in Key Protect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-kpsetup) about how to store the password securely. This option will be deprected in the future.





 Custom API key for each data resource
How IBM Cloud IAM works 

Learn about what IBM Cloud Identity and Access Management (IAM) is, how IAM works, what features are available, and how to access the console, CLI, and APIs to work with IAM in your account.

IAM enables you to securely authenticate users for platform services and control access to resources consistently across IBM Cloud. A set of IBM Cloud services is enabled to use IBM Cloud IAM for access control, and are organized into [resource groups](https://cloud.ibm.com/docs/account?topic=account-rgs) within your account so you can give users access quickly to more than one resource at a time. Each of these services is labeled as "IAM-enabled" in the catalog. You can use IAM access policies to assign users, service IDs, and trusted profiles access to resources within your account. And, you can group users, service IDs, and trusted profiles into an [access group](https://cloud.ibm.com/docs/account?topic=account-groups) to easily give all members of the group the same level of access.

You can use [trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile) to automate the grouping and granting of access to users, services, and app identities. By specifying conditions based on SAML attributes for users whose identity is federated from your external identity provider (IdP), users can be granted access to resources without having to be invited to the account if they meet those conditions. For service and app identities, you can define fine-grained authorization for all applications that are running in a compute resource without creating service IDs or managing the API key lifecycle for applications.

Zoom

![IAM access control in an account](https://cloud.ibm.com/docs-content/v1/content/4d8c6eec891cff72b666dc24bd3211810c77fb42/account/images/tp-in-ag-access-diagram.svg)

Figure 1. How IAM access works in an account by using access groups