vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Identity and Access Management (IAM) Platform Provides authorization checks to Continuous Delivery and toolchain resources and actions. Generates and authenticates IAM API keys such as those that are used by the Delivery Pipeline component. 
 Search and tagging Platform Provides cached, high performance lookup of resources. 
 Resource controller Platform Coordinates the provisioning, updating, and de-provisioning of Continuous Delivery and toolchain resources. 
 Usage metering Platform Receives usage metrics from the Continuous Delivery service to report usage and compute bills for customer accounts. 
 IBM Cloud® Kubernetes Service Service Hosts clusters of containers in which the internal microservices of the Continuous Delivery service run. 
 IBM Cloud® Container Registry Service Stores container images that are used by Delivery Pipeline jobs and steps. 
 IBM® Log Analysis Service Collects internal log events from the Continuous Delivery service for use by IBM specialists in maintaining and supporting the service. 
 IBM Cloud Activity Tracker Service Collects action events from the Continuous Delivery service so that customers can audit activities with the Continuous Delivery service and its components. 
 IBM Cloud® App ID Service Manages conversions between IBM Cloud user IDs and user email addresses. 
 IBM Cloud® Internet Services (CIS) Service Provides domain name resolution and traffic routing for the Continuous Delivery service. 
 IBM® Cloudant® for IBM Cloud® Service Serves as the primary database for all structured Continuous Delivery service and customer-owned data. 
 IBM Cloud® Databases for Redis Service Provides a shared in-memory cache service for high-speed data sharing among replicas of the microservices that make up the Continuous Delivery service. 
 IBM Cloud® Messages for RabbitMQ Service Provides reliable delivery of messages between components of the Continuous Delivery service that include, but are not limited to, Delivery Pipeline triggers. 
 IBM Cloud® Databases for PostgreSQL Service Serves as the database for Delivery Pipeline job and step logs. 
 Object Storage Service Stores artifacts that are produced by Delivery Pipeline jobs and steps. Stores backups of Continuous Delivery data. 
 QRadar Service Collects security events such as successful or failed login authorization attempts for use by IBM specialists who maintain and support the service. 
 General Parallel File System (GPFS) Storage Provides secure, high-performance distributed file system storage of Git Repos and Issue Tracking repositories.
System.out.println("Current time: " + LocalDateTime.now());
AmazonS3 cosClient = createClient(apiKey, serviceInstanceId, endpointUrl, location);

listObjects(cosClient, bucketName);
createBucket(cosClient, newBucketName, storageClass);
listBuckets(cosClient);
}

public static AmazonS3 createClient(String apiKey, String serviceInstanceId, String endpointUrl, String location)
{
AWSCredentials credentials = new BasicIBMOAuthCredentials(apiKey, serviceInstanceId);
ClientConfiguration clientConfig = new ClientConfiguration()
.withRequestTimeout(5000)
.withTcpKeepAlive(true);

return AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpointUrl, location))
.withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig)
.build();
}

public static void listObjects(AmazonS3 cosClient, String bucketName)
{
System.out.println("Listing objects in bucket " + bucketName);
ObjectListing objectListing = cosClient.listObjects(new ListObjectsRequest().withBucketName(bucketName));
for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
System.out.println(" - " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");
}
System.out.println();
}
Release notes for Data Engine 

Use these release notes to learn about the latest IBM Cloud® Data Engine that are grouped by date. Release notes are available for a minimum of three years.



 June 2023 

Store passwords and API keys in IBM Cloud® Secrets Manager
: You can now store passwords and API keys as secrets in Secrets Manager.





 March 2023 

Dark mode available in the Data Engine UI
: The Data Engine UI follows your system theme. You can also choose your theme directly in the profile menu.





 January 2023 

Create a new instance with Terraform
: Detailed [documentation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-terraform) is now available on how to create new Data Engine instances with Terraform.





 November 2022 

Read as text
: Files with an unstable schema, for example logs, can be read faster by using STORED AS TEXT. There is no schema inference and values must further be extracted, as explained in [Query data with an unstable schema](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewquery-logs).





 August 2022 

Support for BYOK encryption for table metadata
: Table metadata for tables that are created after 23 August 2022 that is associated with an instance that uses BYOK get encrypted by using IBM® Key Protect for IBM Cloud®. See [Securing your data in Data Engine](https://cloud.ibm.com/docs/sql-query?topic=sql-query-securing-datadata-storage)

Chennai deprecation
: You cannot create new instances in the Chennai region anymore. Existing instances still work but will be fully deprecated on 31 October.





 May 2022 

Rebranding
: IBM Cloud SQL Query was rebranded to IBM Cloud Data Engine.

Hive
: Data Engine provides an external [Hive metastore (HMS) service](https://cloud.ibm.com/docs/sql-query?topic=sql-query-hive_metastore).





 November 2021 

Add columns to Catalog tables
ibmcloud analytics-engine-v3 spark-app submit [--instance-id INSTANCE-ID] [--app APP] [--runtime RUNTIME] [--jars JARS] [--packages PACKAGES] [--repositories REPOSITORIES] [--files FILES] [--archives ARCHIVES] [--name NAME] [--class CLASS] [--arg ARG] [--conf CONF] [--env ENV]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app (string)
: Path of the application to run. Required.

--runtime ([Runtime](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_enginecli-runtime-example-schema))
: Runtime enviroment for applications and other workloads.

--jars (string)
: Path of the jar files containing the application.

--packages (string)
: Package names.

--repositories (string)
: Repositories names.

--files (string)
: File names.

--archives (string)
: Archive Names.

--name (string)
: Name of the application.

--class (string)
: Entry point for a Spark application bundled as a '.jar' file. This is applicable only for Java or Scala applications.
Configuring Analytics Engine instance 

You can configure IBM Analytics Engine instance to connect to the IBM® watsonx.data instance by setting watsonx.data configurations and Spark related configuration as the default configuration for the IBM Analytics Engine instance.

You can configure Analytics Engine instance with default settings in one of the following ways:



* [Configure by using the IBM Cloud® console](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-config-aelh-cons-config-ae).
* [Configure by using the Analytics Engine API](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-config-aelh-api-config-ae).
* [Configure by using the Analytics Engine CLI](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-config-aelh-cli-config-ae).





 Prerequisites 

Ensure you have the following instances ready:



* IBM® watsonx.data instance.
* IBM Analytics Engine instance.



Fetch the following information from IBM® watsonx.data:



* HMS URL from watsonx.data.For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).
* HMS Credentials from watsonx.data. For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).







 Configuring Analytics Engine instance by using IBM Cloud® console 

To configure your Analytics Engine instance from the IBM Cloud® Resource list, complete the following steps:



1. Log in to your IBM Cloud® account.
2.
* UI
* CLI
* API







 Configure the instance metadata service 

Configure the metadata service by obtaining an instance identity access token from the metadata service. Optionally, generate an IAM access token from this token to access IAM-enabled services in the account.



 Accessing the metadata service by using the instance identity access token service 

To access the instance metadata service, you must first obtain an instance identity access token (a JSON Web Token). You can later generate an IAM token from the instance identity access token and then use it to access IAM-enabled services.

Windows users have extra requirements to set up the metadata service. For more information, see [Setting up Windows servers for using the metadata service](https://cloud.ibm.com/docs/vpc?topic=vpc-imd-windows-configuration).



 Instance identity access token concepts 

An instance identity access token provides a security credential for accessing the metadata service. It's a signed token with a set of claims based on information about the instance and information that is passed in the token request. The minimum version date to use the instance identity access token feature is 2022-03-01.

To access the instance identity, make a PUT "https://api.metadata.cloud.ibm.com/instance_identity/v1/token call by using the [Metadata service API](https://cloud.ibm.com/apidocs/vpc-metadatacreate-access-token) that invokes the instance host name. Communication between the instance and metadata service never leaves the host, you acquire the token from within the instance. If https secure protocol is not enabled on your instance, you can use your instance IP address instead of the host name.

In the request, you specify an expiration time for the token. The default is 5 minutes, but you can specify that it expires sooner or later (5 seconds to 1 hour).

The response (a JSON payload) contains the instance identity access token. Use this token to access the metadata service.

You can also generate an IAM token from this token and use the RIAS API to call IAM-enabled services.
NLU voyager-nlu-slot-${release-name} 
 Master voyager-master-slot-${release-name} 
 TAS tas-runtime-slot-${release-name} 
 ED-MM ${release-name}-ed-mm 



Microservice pods that are either LiteLinks clients or servers contain an initContainer that, on pod creation, checks, and waits until Etcd is operational and contains configuration entries.





 Configuration storage 

The microservices in the language understanding pipeline use Etcd to store some configuration values.

Each microservice has its own path in Etcd. Other metadata about models, such as the instance in which a model is loaded, is stored under different keys under the Etcd path.



Microservice configuration store Etcd paths

 Microservice Etcd path 

 NLU /bluegoat/voyager-nlu/voyager-nlu-slot-${release-name} 
 Master /bluegoat/bluegoat-master/voyager-master-slot-${release-name} 
 TAS /bluegoat/tas-runtime/tas-runtime-slot-${release-name} 
 ED-MM /bluegoat/tas-runtime/${release-name}-ed-mm/ 



The configuration values per microservice are stored under a /config subpath.







 MinIO 

MinIO is an enterprise-grade object storage service that implements the Amazon S3 API. The Watson Assistant chart runs MinIO in distributed mode with four pods. With this configuration, MinIO functions fully in read/write mode if more than half, meaning three or more, of the pods are available. Its function is degraded to read-only mode if only half (two) of the pods are available.

MinIO pods are named ${release-name}-clu-minio-[0-9]. CLU, which stands for Conversational Language Understanding, is included in the pod names to associate the MinIO pods with the language understanding pipeline.
[IBM Cloud® Databases for Redis](https://cloud.ibm.com/catalog/services/databases-for-redis) databases-for-redis 
 [IBM Cloud® Databases for MySQL](https://cloud.ibm.com/catalog/services/databases-for-mysql) databases-for-mysql 
 [IBM Cloud® Messages for RabbitMQ](https://cloud.ibm.com/catalog/services/messages-for-rabbitmq) messages-for-rabbitmq 



IBM Cloud® Databases for DataStax is deprecated and no longer supported as of 30 June 2024. For more information, see the [deprecation details](https://cloud.ibm.com/docs/databases-for-cassandra?topic=databases-for-cassandra-deprecationdep_details).



 Using the catalog 

When you create the deployment from the catalog, specify the following parameters.



Required

The service name - The name can be any string and is the name that is used on the web and in the CLI to identify the new deployment.
Required

Location - The deployment's public cloud region or Satellite location.
Required

Database Version - The deployment version of your database. To ensure optimal performance, run the preferred version. The latest minor version is used automatically.
Optional

The resource group - If you are organizing your services into [resource groups](https://cloud.ibm.com/docs/account?topic=account-account_setup), specify the resource group in this field. Otherwise, you can leave it at default.
Optional

Key Protect instance and disk encryption key - If you use Key Protect, an instance and key can be selected to encrypt the deployment's disk. If you do not use your own key, the deployment automatically creates and manages its own disk encryption key.
Optional

Initial resource allocation - Specify initial memory and disk sizes for your databases. The minimum sizes of memory and disk are selected by default.
Optional

CPU allocation - Choose dedicated compute resources for your deployment. With dedicated cores, your resource group is given a single-tenant host with a minimum reserve of cpu shares.
import com.ibm.cloud.objectstorage.services.s3.AmazonS3ClientBuilder;
import com.ibm.cloud.objectstorage.services.s3.model.Bucket;
import com.ibm.cloud.objectstorage.services.s3.model.BucketLifecycleConfiguration;
import com.ibm.cloud.objectstorage.services.s3.model.ListObjectsRequest;
import com.ibm.cloud.objectstorage.services.s3.model.ObjectListing;
import com.ibm.cloud.objectstorage.services.s3.model.S3ObjectSummary;
import com.ibm.cloud.objectstorage.oauth.BasicIBMOAuthCredentials;

public class App
{
private static AmazonS3 _cosClient;

/
* @param args
/
public static void main(String[] args)
{
SDKGlobalConfiguration.IAM_ENDPOINT = "https://iam.cloud.ibm.com/identity/token";
String bucketName = "<sample-bucket-name>";
String api_key = "ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE";
String service_instance_id = "85SAMPLE-eDOb-4NOT-bUSE-86nnnb31eaxx"
String endpoint_url = "https://s3.us-south.cloud-object-storage.appdomain.cloud";

String storageClass = "us-south";
String location = "us";

_cosClient = createClient(api_key, service_instance_id, endpoint_url, location);

// Use the client to read the configuration
BucketLifecycleConfiguration config = _cosClient.getBucketLifecycleConfiguration(bucketName);

System.out.println(config.toString());
}

/
* @param bucketName
It can connect to only one regional endpoint (home). For example, if an agent is deployed on a cluster in Sydney and has been configured to use the Schematics eu-de regional endpoint as it’s home location. The agent polls for jobs in eu-de region. Hence, the workspace or action to deploy resources using the Sydney agent must be created in the eu-de region.





 Can I register an agent with multiple accounts? 

No, you cannot register an agent with multiple accounts in the beta release.





 Can jobs for an existing workspace be configured to run on an agent? 

Yes, if your workspace has the right values with the tags, resource-group, location. Schematics uses an agent-selection-policy to automatically assign the jobs to run on the target agent.

For example, If you have an existing workspace: wks-0120 with tag=dev, and you want the workspace to run on Agent-1. Create an agent-selection-policy with the rule to pick Agent-1 when the tag == dev. Subsequently, the workspace job such as plan, apply, update, and so on will be dynamically routed to Agent-1.





 What IAM permissions needed to deploy an agent? 

For information about identity and permissions, see [agent permission](https://cloud.ibm.com/docs/schematics?topic=schematics-accessagent-permissions).





 Can I use Terraform custom providers or use a proxy registry to download Terraform provider plug-ins? 

Agents supports the use of custom Terraform providers sourced from a private Terraform registry with Schematics Terraform jobs. The support to use custom providers is not available in the shared multi-tenant Schematics service. It is only available with Agents. Agents does not include a local or private provider registry. The registry must be provided and configured by the user on the users private network accessible to the agents.

By default, Schematics jobs running the Terraform CLI will download Terraform provider plug-ins or Terraform modules from the public Terraform registry via the Internet or public network.
* Fixes EADDRINUSE error caused by listeners that do not execute a teardown appropriately when a destination is deleted.
* Resolve issue where service instances bound to an application could not be deleted.









 v1.7.0 

Published date: 2017-03-16



 Features 



* Introduces new service plans: Essentials, Professional, and Enterprise.
* The Secure Gateway Client now supports SOCKS proxy between itself and the on-prem destination.









 v1.6.1 

Published date: 2017-02-03



 Fixes 



* Resolve issue where disconnecting multiple clients results in orphaned processes in the connected clients array.
* Now cleans up orphaned connections that resulted in incorrecty paused listeners.









 v1.6.0 

Published date: 2016-10-06



 Features 



* Access Control List now supports specific path routing for HTTP/S requests.
* Destinations now support Server Name Indicators for TLS connections.









 v1.5.1 

Published date: 2016-08-10



 Features 



* Destination wizard added for destination creation.
* Gateways now support uploading a custom cert/key pair.
* Services are now limited to 50 gateways and 50 destinations per gateway.
* Destinations/Gateways/Services can now be exported/imported.
* Latency tests between client and server capable of being executed from Secure Gateway UI.
* Cleans up orphaned tunnel processes on network disconnect.
* Resolve duplicate port allocation caused by failed destination deletion.
* Resolve HTTP/S issue with non-UTF8 encoding.
* Resolve missing IPC handlers on replacement processes.









 v1.5.0 

Published date: 2016-05-26



 Features 



* Client now has a local interactive UI.
* UDP connections are now supported.
* Memory footprint of the client has been drastically reduced.
* Single client mode is no longer supported; multi-client will be default and transparent to a single-gateway client.
* Access Control List synchronizes between clients connected to the same gateway.









 v1.4.2 

Published date: 2016-03-21



 Features 



* Clients are now assigned an ID when connecting to the SG server.
* Communication with the client(s), for example SAP ABAP application servers or SAP HANA Studio for administration purposes.



Use the network that holds the default route of your environment to pass the NFS traffic through it, the storage servers are reachable through that gateway, only.





 Storage for Scale-out configurations 

For scale-out configuration, the ability of storage volumes to be accessed from different server nodes is required for failover purposes. Thus, local storage is out of scope, and NFS volumes need to be deployed. The deployed volumes can vary in size and number (see details here: [Persistent Data Storage in the SAP HANA Database](https://help.sap.com/docs/SAP_HANA_PLATFORM/6b94445c94ae495c83a19646e7c3fd56/be3e5310bb571014b3fbd51035bc2383.html)). In any case, they have to comply with the TDI performance KPIs (see [SAP Note 2613646](https://launchpad.support.sap.com//notes/2613646)) verified by [SAP HANA Hardware and Cloud Measurement Tools](https://help.sap.com/docs/HANA_HW_CLOUD_TOOLS/02bb1e64c2ae4de7a11369f4e70a6394/7e878f6e16394f2990f126e639386333.html).

IBM Cloud recommends Endurance File Storage at 10 IOPS per GB or Performance File Storage with IOPS equal or greater than 10K. For the network configuration, use the primary network as storage network to guide the traffic to the NFS servers through it.







 Intel Virtual Servers in VPC Infrastructure (Gen2) 



 Available SAP HANA certified IBM Cloud configurations 

For Intel-based VSIs in VPC, the following configuration is available for OLAP scale-out configuration with SAP HANA:

OLAP:



* vx2-176x2464
The scripts are designed to create a new VPC and install SAP (SAP NW 7.x release) solution together with its dedicated DB SAP HANA box in one task flow.





 SAP Kits 

For each IBM Cloud region, IBM allocates temporary storage on a dedicated temporary Deployment server (Bastion Server) that is used for terraform environment. It is your responsibility to download the necessary SAP and DB kits to your Deployment (Bastion) Server. All file archives are decompressed by Ansible during the automatic deploying process. For more information, see the readme file in the dedicated GitHub repository.





 Terraform deployment 

The configuration and script files are provided on the GitHub repository:



* For JAVA stack : [https://github.com/IBM-Cloud/sap-netweaver-java-hana](https://github.com/IBM-Cloud/sap-netweaver-java-hana).
* For ABAP stack : [https://github.com/IBM-Cloud/sap-netweaver-abap-hana](https://github.com/IBM-Cloud/sap-netweaver-abap-hana)



For SAP HANA stand-alone virtual server instance on IBM Cloud Virtual Private Cloud, you modify the:



* The input.auto.tfvars file to customize the resources for your solution. You specify zones, resource names, SSH keys, and SAP variables.



All of the other configuration files are provided and do not need to be modified.





 Schematics deployment 

The configuration and script files are provided on the GitHub repository:



* ABAP: [https://github.com/IBM-Cloud/sap-netweaver-abap-hana/tree/main/schematics](https://github.com/IBM-Cloud/sap-netweaver-abap-hana/tree/main/schematics)