vania-janet/multiturn-rag-retrieval-data huggingface dataset:
IBM Analytics Engine instance administration <br><br> * Provide infrastructure operating system (OS), version, and security updates.<br> * Clean up all instance resources.<br> * Track hardware issues on running cluster.<br><br><br> <br><br> * Create an instance using the provided API, CLI or console tools.<br> * Delete a service instance using the provided API, CLI or console tools.<br> * Customize a service instance using the provided API or CLI.<br> * View or change the instance configuration using the provided API, CLI or console tools.<br><br><br> 
 Application administration <br><br> * Monitor Spark application for any failures due to infrastructure provided by IBM.<br><br><br> <br><br> * Run Spark applications on the cluster using the provided CLI or API.<br> * Tune the Spark instance for your application requirements using the provided CLI or API.<br><br><br> 
 Observability <br><br> * Provide Log Analysis to enable observability of your IBM Analytics Engine service logs.<br> * Provide integration with Activity Tracker to send IBM Analytics Engine events for auditability.<br><br><br> <br><br> * Set up Activity Tracker and send events to monitor the health of your IBM Analytics Engine instances.<br> * Set up and send logs to Log Analysis.<br><br><br> 







 Change management 

Change management includes tasks such as deployment, configuration, upgrades, patching, configuration changes, and deletion.



Table 2. Responsibilities for change management
The rows are read from left to right. The first column describes the task that a the customer or IBM might be responsibility for. The second column describes IBM responsibilities for that task. The third column describes your responsibilities as the customer for that task.

 Task IBM Responsibilities Your Responsibilities
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server stop 

Stop the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server stop [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 history-server stop --output json

ibmcloud analytics-engine-v3 history-server stop --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json









 Log configuration 

Note: Deprecated. Use equivalent commands from the [log-forwarding-config](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-log-forwarding-config-cli) command group instead.

ibmcloud analytics-engine-v3 log-config --help



 ibmcloud analytics-engine-v3 log-config update 

Enable or disable log forwarding from IBM Analytics Engine to IBM Log Analysis server.

Note: Deprecated. Use replace from the log-forwarding-config command group instead.
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to stop a submitted application ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to view all submitted Spark applications ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to view a specific Spark application, by using the application ID !
curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enabled":true,"sources": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '
* By using the logging API endpoint (deprecated):

curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enable":true,"components": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '









 Disabling platform logging from IBM Analytics Engine 

Disable forwarding IBM Analytics Engine platform logs to IBM Log Analysis, in one of the following ways.



* By using the log_forwarding_config API endpoint:

curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":false}'
* By using the logging API endpoint (deprecated):

curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":false}'







 Viewing the log forwarding configuration of an IBM Analytics Engine instance
HMS Overview 



  Hive Metastore 

Hive Metastore (HMS) is a service that stores metadata related to Presto and other services in a backend Relational Database Management System (RDBMS) or Hadoop Distributed File System (HDFS).

When you create a new table, information related to the schema such as column names, data types etc is stored in the metastore relational database. A metastore enables the user to see the data files in the HDFS object storage as if they are stored in tables with HMS.

Metastore acts as a bridge between the schema of the table and the data files stored in object storages. HMS holds the definitions, schema, and other metadata for each table and maps the data files and directories to the table representation which is viewed by the user. Therefore, HMS is used as a storage location for the schema and tables. HMS is a metastore server that connects to the object storage to store data and keeps its related metadata on PostgreSQL.

Any database with a JDBC driver can be used as a metastore. Presto makes requests through thrift protocol to HMS. The Presto instance reads and writes data to HMS. HMS supports 5 backend databases as follows. In IBM® watsonx.data, PostgreSQL database is used.



*  Derby
*  MySQL
*  MS SQL Server
*  Oracle
*  PostgreSQL



Currently HMS in watsonx.data supports Iceberg table format.

Following three modes of deployment are supported for HMS. In watsonx.data the remote mode is used.



*  Embedded Metastore - Derby with singe session.
*  Local Metastore - MySQl with multiple session accessible locally.
*  Remote Metastore - metastore runs on its own separate JVM and accessible via thrift network APIs.
Download the [Hive-compatible client](https://us.sql-query.cloud.ibm.com/download/catalog/hive-metastore-standalone-client-3.1.2-sqlquery-1.0.13.jar) and place it in a directory of your Apache Spark cluster that is not on the classpath. This step is necessary, as the client is loaded into an isolated classloader to avoid version conflicts. Note that in the examples the files are placed in /tmp/dataengine, when you use a different folder, adjust the example accordingly.

The client differs from the Hive version 3.1.2 release by more enhancements that add support for TLS and authentication through IBM Cloud® Identity and Access Management. For user, specify the CRN and for password a valid API key with access to your Data Engine. Find the endpoint to use in the following table.



Table 1. Region endpoints

 Region Endpoint 

 us-south thrift://catalog.us.dataengine.cloud.ibm.com:9083 
 eu-de thrift://catalog.eu-de.dataengine.cloud.ibm.com:9083 







 Convenience libraries to configure Spark 

While the Data Engine catalog is compatible with the Hive metastore and can be used as any other external Hive metastore server, an SDK is provided to minimize the steps that are needed to configure Apache Spark. The SDK simplifies connecting to the Hive metastore and IBM Cloud Object Storage buckets referenced by tables or views.

In case of using Python download both, the Scala and the Python SDK, and place them in a folder that is in the classpath of your Apache Spark cluster. When using Scala, the Scala SDK is enough.



* [spark-dataengine-scala](https://us.sql-query.cloud.ibm.com/download/catalog/dataengine-spark-integration-1.4.51.jar)
Integrating Presto with Apache Hudi using a Hudi connector 

You can integrate Presto with Apache Hudi by using the Hudi connector. You can query Hudi tables that are synced to Hive metastore (HMS) using Presto's SQL interface. This combination offers the benefits of fast and interactive analytics on large-scale, high-velocity data stored in Hudi. Hudi connector uses the metastore to track partition locations. It uses underlying Hudi file system and input formats to list data files.



  Configuring a catalog in Presto 

Create a hudi.properties file inside /opt/presto/etc/catalog directory in the presto container.

 hudi.properties
connector.name=hudi

 HMS thrift URI
hive.metastore.uri=thrift://<hostname>:<port>

 properties to enable connection to object-storage bucket
hive.s3.ssl.enabled=true
hive.s3.path-style-access=true
hive.s3.endpoint=<Bucket API Endpoint>
hive.s3.aws-access-key=<INSERT YOUR ACCESS KEY>
hive.s3.aws-secret-key=<INSERT YOUR SECRET KEY>

 properties to enable TLS connection to HMS
hive.metastore.thrift.client.tls.enabled=true
hive.metastore.authentication.type=PLAIN
hive.metastore.thrift.client.tls.truststore.path=<Truststore Path>
hive.metastore.thrift.client.tls.truststore.password=<Truststore Password>
hive.metastore.thrift.client.tls.keystore.path=<Keystore Path>
hive.metastore.thrift.client.tls.keystore.password=<Keystore Password>
Show more





  Limitations 



1.  Connector does not support DDL or DML SQL statements. Presto can query data using the Hudi connector, but cannot directly perform write operations.
2.  Data modifications must be done through Hudi-specific tools and workflows.
Note that the library set name certificate_library_set must match the value of the Databases for PostgreSQL metastore connection parameter ae.spark.librarysets that you specified.



4. Specify the following Databases for PostgreSQL metastore connection parameters as part of the Spark application payload or as instance defaults. Make sure that you use the private endpoint for the "spark.hadoop.javax.jdo.option.ConnectionURL" parameter below:

"spark.hadoop.javax.jdo.option.ConnectionDriverName": "org.postgresql.Driver",
"spark.hadoop.javax.jdo.option.ConnectionUserName": "ibm_cloud_<CHANGEME>",
"spark.hadoop.javax.jdo.option.ConnectionPassword": "<CHANGEME>",
"spark.sql.catalogImplementation": "hive",
"spark.hadoop.hive.metastore.schema.verification": "false",
"spark.hadoop.hive.metastore.schema.verification.record.version": "false",
"spark.hadoop.datanucleus.schema.autoCreateTables":"true",
"spark.hadoop.javax.jdo.option.ConnectionURL": "jdbc:postgresql://<CHANGEME>.databases.appdomain.CHANGEME/ibmclouddb?sslmode=verify-ca&sslrootcert=/home/spark/shared/user-libs/certificate_library_set/custom/postgres.cert&socketTimeout=30",
"ae.spark.librarysets":"certificate_library_set"
5. Set up the Hive metastore schema in the Databases for PostgreSQL instance because there are no tables in the public schema of Databases for PostgreSQL database when you create the instance. This step executes the Hive schema related DDL so that metastore data can be stored in them.
spark.sql.catalog.lakehouse.uri = <hms-thrift-endpoint-from-watsonx.data> for example (thrift://81823aaf-8a88-4bee-a0a1-6e76a42dc833.cfjag3sf0s5o87astjo0.databases.appdomain.cloud:32683)
spark.hive.metastore.client.auth.mode = PLAIN
spark.hive.metastore.client.plain.username = <hms-user-from-watsonx.data> (for example, ibmlhapikey)
spark.hive.metastore.client.plain.password = <hms-password-from-watsonx.data>
spark.hive.metastore.use.SSL = true
spark.hive.metastore.truststore.type = JKS
spark.hive.metastore.truststore.path = file:///opt/ibm/jdk/lib/security/cacerts
spark.hive.metastore.truststore.password = changeit
Show more



Parameter value:



* Hms-thrift-endpoint-from-watsonx.Data: Specify the credentials for watsonx.data.
* Hms-user-from-watsonx.Data: The watsonx.data username.
* Hms-password-from-watsonx.Data: The watsonx.data password.







 Configuring Analytics Engine instance by using Analytics Engine API 

To configure your IBM Analytics Engine instance from the Analytics Engine API, complete the following steps:



1. Generate an IAM token to connect to the IBM Analytics Engine API. For more information about how to generate an IAM token, see [IAM token](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverlessendpoints-cli).
2. Run the following API command to invoke the Analytics Engine API by using the generated IAM token.
You can then configure your IBM Analytics Engine instance to use the default metastore configuration either at instance level or at application level as needed.

IBM Cloud Data Engine supports creating instances for different endpoints(location). Within an instance, different IBM Cloud Object Storage buckets are created to store data. The data buckets can be created for different end points(region). The endpoints for the data engine instance(thrift) and the data bucket are different. Ensure that you select the correct endpoints that are supported by the system.
* For more information about the applicable endpoint(thrift) for your region while creating instance, see [Thrift endpoint](https://cloud.ibm.com/docs/sql-query?topic=sql-query-hive_metastorehive_compatible_client).
* For more information on the currently supported data engine endpoints, see [Data engine endpoints](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewendpoints)
{: important}
Show more





 Storing data in Cloud Object Storage 

Generate and store data in cloud object storage. Run the following regular PySpark application, called generate-and-store-data.py in this example, which stores Parquet data in some location on IBM Cloud Object Storage.

Example
: Enter:

 {: codeblock}
from pyspark.sql import SparkSession

def init_spark():
    spark = SparkSession.builder.appName("dataengine-generate-store-parquet-data").getOrCreate()
    sc = spark.sparkContext
    return spark,sc

def generate_and_store_data(spark,sc):
    data =[("India","New Delhi"),("France","Paris"),("Lithuania","Vilnius"),("Sweden","Stockholm"),("Switzerland","Bern")]
    columns=["Country","Capital"]
    df=spark.createDataFrame(data,columns)
You can monitor and manage your clusters by using the tools available in Ambari or additional services provided by IBM Analytics Engine.
* Where are my job log files?

For most components, the log files can be retrieved by using the Ambari GUI. Navigate to the respective component, click Quick Links and select the respective component GUI. An alternative method is to SSH to the node where the component is running and access the /var/log/<component> directory.
* How can I debug a Hive query on IBM Analytics Engine?

To debug a Hive query on IBM Analytics Engine:



1. Open the Ambari console, and then on the dashboard, click Hive > Configs > Advanced.
2. Select Advanced > hive-log4j and change hive.root.logger=INFO,RFA to hive.root.logger=DEBUG,RFA.
3. Run the Hive query.
4. SSH to the IBM Analytics Engine cluster. The Hive logs are located in /tmp/clsadmin/hive.log.



* What type of encryption is supported?

All data on IBM Cloud Object Storage is encrypted at-rest. You can use a private, encrypted endpoint available from IBM Cloud Object Storage to transfer data between IBM Cloud Object Storage and IBM Analytics Engine clusters. Any data that passes over the public facing ports (8443,22 and 9443) is encrypted. See details in [Best practices](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-best-practicescos-encryption).
* Which ports are open on the public interface on the cluster?

The following ports are open on the public interface on the cluster:



* Port 8443 Knox
* Port 22 SSH
* Port 9443 Ambari



* What is IBM Analytics Engine serverless?

The IBM Analytics Engine Standard serverless plan for Apache Spark offers a new consumption model using Apache Spark. An Analytics Engine serverless instance does not consume any resources when no workloads are running. When you submit Spark applications, Spark clusters are created in seconds and are spun down as soon as the applications finish running.
-- set the index location for the table CUSTOMERS_PARTITIONED
ALTER TABLE CUSTOMERS_PARTITIONED DROP METAINDEX LOCATION







 IndexAsset 



 metaindexAsset 

The indexAsset is either based on a table or Cloud Object Storage location.

The metaindexAssetLocation is a subset of the [externalTableSpec](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec).

The metaindexAssetHiveTable refers to a Hive table.









 Miscellaneous definitions 



 COSURI 

A Cloud Object Storage Uniform Resource Identifier (COS URI) is a string of characters that uniquely identifies an object on Cloud Object Storage. By definition URIs are case-insensitive.

The syntax of a Cloud Object Storage URI is thoroughly described in section [Table unique resource identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewunique).





 CRN_URI 

A database table CRN is a unique identifier that consists of the CRN of a database service instance and a specific table name that instance. The user must have access to this service instance and its credentials.

The syntax of a table CRN is thoroughly described in section [Table unique resource identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewunique).





 DB2_TABLE_URI 

A Db2 table URI is a string of characters that uniquely identifies a table in an IBM® Db2® on Cloud and IBM® Db2® Warehouse on Cloud instance. The instance must be enabled for IAM and the IBMid if the user must be added as a database user.

The syntax of a Db2 Table URI is thoroughly described in section [Table unique resource identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewunique).





 Identifier 

An identifier is a name that uniquely identifies an entity. The two types of identifiers are unquoted identifiers and back quoted identifiers.
source-s3-creds S3 credentials of data source source-s3-creds:<SOURCE_S3_CREDS> 
 staging-location Location where CSV files are converted to Parquet are staged. Even parquet files in some circumstances are staged like, Parquet file is local, Parquet files in local folder, a huge Parquet file(local or S3) all of which may not fit in memory, source file is S3 parquet file, and file from S3 parquet folder which may contain special column types such as date or decimal. Example: <s3://lh-target/staging> staging-location:<STAGING_LOCATION> 
 staging-s3-creds S3 credentials of staging location source-s3-creds:<SOURCE_S3_CREDS> 
 staging-hive-catalog If the default hive catalog cannot be used due to credential mismatch with source or staging locations, it is possible to configure a custom hive catalog using this option staging-hive-catalog:<catalog_name> 
 system-config This parameter can be used to specify system related parameters such as memory limit system-config:<file_path> 
 target-catalog-uri Target catalog uri target-catalog-uri:<TARGET_CATALOG_URI> 
 target-table-storage Target table file storage location target-table-storage:<TARGET_TABLE_STORAGE> 
 target-tables Data migration target table. <catalog>.<schema>.<table1>. Example: <iceberg.demo.customer1> target-tables:<TARGET_TABLES> 
 schema Schema file that includes CSV specifications, and so on schema:<SCHEMA> 
 target-s3-creds S3 credentials of target location target-s3-creds:<TARGET_S3_CREDS> 
 ingestion-engine-endpoint Endpoint of ingestion engine. hostname=<hostname>,port=<port> ingestion-engine-endpoint:<INGESTION_ENGINE_ENDPOINT> 
 dbuser Username of ingestion engine dbuser:<DBUSER>
_cosClient = createClient(api_key, service_instance_id, endpoint_url, location);

// Use the client to read the configuration
BucketLifecycleConfiguration config = _cosClient.getBucketLifecycleConfiguration(bucketName);

System.out.println(config.toString());
}

/
* @param bucketName
* @param clientNum
* @param api_key
* @param service_instance_id
* @param endpoint_url
* @param location
* @return AmazonS3
/
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}
Show more





 Delete a bucket’s lifecycle configuration, including expiration 

This implementation of the DELETE operation uses the lifecycle query parameter to examine lifecycle settings for the bucket. All lifecycle rules associated with the bucket will be deleted. Transitions defined by the rules will no longer take place for new objects. However, existing transition rules will be maintained for objects that were already written to the bucket before the rules were deleted. Expiration Rules will no longer exist.
When the staging location is not used, ensure that the hive catalog configured with Presto can be used with source-files location. Following are the exception cases where staging location is required:



* Any or all of the parquet files in the folder are huge.
* Any or all of the parquet files in the folder have special columns, such as date or decimal.







 Ingest a CSV/Parquet file or a folder of files from a local file system by using config file 

To ingest a CSV/Parquet file or a folder of files from a local file system, update the following configuration file:

[global-ingest-config]
target-tables:<TARGET_TABLES>
ingestion-engine:hostname=<hostname>,port=<port>
create-if-not-exist:<true>
[ingest-config1]
source-files:<SOURCE_DATA_FILE>
staging-location:<STAGING_LOCATION>
staging-s3-creds:AWS_SECRET_ACCESS_KEY=<YOUR_TARGET_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_TARGET_ACCESS_KEY_ID>,AWS_REGION=<YOUR_TARGET_REGION>, BUCKET_NAME=<YOUR_TARGET_BUCKET>, ENDPOINT_URL=<YOUR_TARGET_ENDPOINT_URL>

For example:

[global-ingest-config]
target-tables:iceberg_cat.ice_schema.customer1_tab
ingestion-engine:hostname=localhost,port=8080
create-if-not-exist:true

[ingest-config1]
source-files:/tmp/customer1.parquet
staging-location:s3://cust-bucket/warehouse/staging/