vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Manage Spark history server Administrator <br>Developer ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Access Spark history Administrator <br>Developer <br>DevOps ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) !
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 View list of submitted Spark applications Administrator <br>Developer <br>DevOps Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Stop submitted Spark applications Administrator <br>Developer <br>DevOps Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Customize libraries Administrator <br>Developer Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) Not available 
 Access job logs Administrator <br>Developer <br>DevOps !
applications_api: https://api-staging.us-south.ae.cloud.ibm.com/v3/analytics_engines/xxxx660-7f2x-4xxf-9xx4-exxxxxxxxxc7/spark_applications
instance_api: https://api-staging.us-south.ae.cloud.ibm.com/v3/analytics_engines/xxxx660-7f2x-4xxf-9xx4-exxxxxxxxxc7
history_server_ui: https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/xxxx660-7f2x-4xxf-9xx4-exxxxxxxxxc7/spark_history_ui
history_server_api: https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/xxxx660-7f2x-4xxf-9xx4-exxxxxxxxxc7/spark_history_api/v1

iam_apikey_description: Auto-generated for key 4f1xxx3c-7517-xxx-cbfxxxxxx5d0
iam_apikey_name: service-key-name
iam_role_crn: crn:v1:bluemix:public:iam::::serviceRole:Writer
iam_serviceid_crn: crn:v1:staging:public:iam-identity::a/xxxx::serviceid:ServiceId-3543db7f-xxxx-454a-xxxx-1b20333ebeaa
Show more

To view your service resource key, enter the following command:

ibmcloud resource service-key <service_key_name>

Required parameter:



* <service key name>: the name of the service key that you entered when creating the key.



Sample Response: Bear in mind that the cluster credentials are not returned in the response.

{
"apikey": "xxxxx",
"endpoints": {
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Configure logging Administrator <br>Developer <br>Devops ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) Not available 
 Submit Spark applications Administrator <br>Developer Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 View list of submitted Spark applications Administrator <br>Developer <br>DevOps Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) !
ServiceCall<ApplicationCollection> listApplications(ListApplicationsOptions listApplicationsOptions)

Example request:

// Construct an instance of the ListApplicationsOptions model
ListApplicationsOptions listApplicationsOptionsModel = new ListApplicationsOptions.Builder()
.instanceId("dc0e9889-eab2-4t9e-9441-566209499546")
.build();

// Invoke operation with valid options model (positive test)
Response<ApplicationCollection> response = ibmAnalyticsEngineApiService.listApplications(listApplicationsOptionsModel).execute();
ApplicationCollection responseObj = response.getResult();
System.out.println(String.valueOf(responseObj));
* Retrieve the details of a given Spark application:

ServiceCall<ApplicationGetResponse> getApplication(GetApplicationOptions getApplicationOptions)

Example request:

// Construct an instance of the GetApplicationOptions model
GetApplicationOptions getApplicationOptionsModel = new GetApplicationOptions.Builder()
.instanceId("dc0e9889-eab2-4t9e-9441-566209499546")
.applicationId("db933645-0b68-4dcb-80d8-7b71a6c8e542")
.build();

// Invoke operation with valid options model (positive test)
Response<ApplicationGetResponse> response = ibmAnalyticsEngineApiService.getApplication(getApplicationOptionsModel).execute();
ApplicationGetResponse responseObj = response.getResult();
System.out.println(String.valueOf(responseObj));
* Stop a running application identified by the app_id identifier.
The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app-id (string)
: Identifier of the application for which details are requested. Required.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 spark-app status --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b --output json

ibmcloud analytics-engine-v3 spark-app status --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b --output json





 Example output 

The output is as follows:

{
"id" : "9da32aaf-df69-4e61-bdb8-1b2772c0f677",
"state" : "finished",
"start_time" : "2021-04-21T04:24:01Z",
"end_time" : "2021-04-21T04:28:15Z",
"finish_time" : "2021-04-21T04:28:15Z",
"auto_termination_time" : "2021-04-24T04:24:01Z"
}
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Customize libraries Administrator <br>Developer Not available ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) Not available 
 Access job logs Administrator <br>Developer <br>DevOps ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) from the Log Analysis console Not applicable Not applicable 
 View instance details; shown details might vary depending on access role Administrator <br>Developer <br>DevOps ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Manage Spark history server Administrator <br>Developer !
func main() {
// Construct an instance of the ListApplicationsOptions model
listApplicationsOptionsModel := new(ibmanalyticsengineapiv3.ListApplicationsOptions)
listApplicationsOptionsModel.InstanceID = core.StringPtr("dc0e9889-eab2-4t9e-9441-566209499546")

_, applicationCollection, _ := ibmAnalyticsEngineApiService.ListApplications(listApplicationsOptionsModel)
fmt.Println(applicationCollection)
}
* Retrieve the details of a given Spark application:

(ibmAnalyticsEngineApi IbmAnalyticsEngineApiV3) GetApplication(getApplicationOptions GetApplicationOptions) (result ApplicationGetResponse, response core.DetailedResponse, err error)

Example request:

func main() {
// Construct an instance of the GetApplicationOptions model
getApplicationOptionsModel := new(ibmanalyticsengineapiv3.GetApplicationOptions)
getApplicationOptionsModel.InstanceID = core.StringPtr("dc0e9889-eab2-4t9e-9441-566209499546")
getApplicationOptionsModel.ApplicationID = core.StringPtr("db933645-0b68-4dcb-80d8-7b71a6c8e542")

_, applicationGetResponse, _ := ibmAnalyticsEngineApiService.GetApplication(getApplicationOptionsModel)
fmt.Println(applicationGetResponse)
}
* Stop a running application identified by the app_id identifier. This is an idempotent operation. Performs no action if the requested application is already stopped or completed.
About email delivery 

SendGrid, IBM Cloud® email delivery service for Classic infrastructure, is a mail relay service. With this service, you can use a smart host to relay your outbound mail services. A smart host relays SMTP traffic from an SMTP server, mail client, or any other service or programming language capable of handling SMTP. This service also generates metrics, tracks email lists, and tracks email activity such as bounced, clicked, dropped, and opened emails. Other available features are offered with this service such as newsletter assistance and authentication.
a 5 17.0
a 3 7.0
b 2 45.0

Create a timeseries from a dataframe, providing a timestamp and a value column
>>> ts = tspy.time_series(df, ts_column="timestamp", value_column="value")
>>> ts
TimeStamp: 1 Value: 27.0
TimeStamp: 2 Value: 45.0
TimeStamp: 3 Value: 4.0
TimeStamp: 3 Value: 7.0
TimeStamp: 5 Value: 17.0
Show more

To revert from a time series back to a pandas DataFrame, use the to_df function:

>>> import sparktspy as tspy
>>> ts_orig = tspy.time_series([1.0, 2.0, 3.0])
>>> ts_orig
TimeStamp: 0 Value: 1
TimeStamp: 1 Value: 2
TimeStamp: 2 Value: 3

>>> df = ts_orig.to_df()
>>> df
timestamp value
0 0 1
1 1 2
2 2 3





 Data model 

Time series data does not have any standards for the model and data types, unlike some data types such as spatial, which are governed by a standard such as Open Geospatial Consortium (OGC). The challenge with time series data is the wide variety of functions that need to be supported, similar to that of Spark Resilient Distributed Datasets (RDD).

The data model allows for a wide variety of operations ranging across different forms of segmentation or windowing of time series, transformations or conversions of one time series to another, reducers that compute a static value from a time series, joins that join multiple time series, and collectors of time series from different time zones. The time series library enables the plug-and-play of new functions while keeping the core data structure unchangeable. The library also support numeric and categorical typed timeseries.
1.0





 sin 

sin(expr)
: Returns the sine of expr.
: Example of an SQL function usage fragment

> SELECT sin(0)
: Result value

0.0





 sinh 

sinh(expr)
: Returns the hyperbolic sine of expr.
: Example of an SQL function usage fragment

> SELECT sinh(0)
: Result value

0.0





 size 

size(expr)
: Returns the size of an array or a map. Returns -1 if null.
: Example of an SQL function usage fragment

> SELECT size(array('b', 'd', 'c', 'a'))
: Result value

4





 skewness 

skewness(expr)
: Returns the skewness value that is calculated from values of a group.





 smallint 

smallint(expr)
: Casts the value expr to the target data type smallint.





 sort_array 

sort_array(array[, ascendingOrder])
: Sorts the input array in ascending or descending order according to the natural ordering of the array elements.
: Example of an SQL function usage fragment

> SELECT sort_array(array('b', 'd', 'c', 'a'), true)
: Result value

["a","b","c","d"]





 soundex 

soundex(str)
: Returns Soundex code of the string.
: Example of an SQL function usage fragment

> SELECT soundex('Miller')
: Result value

M460





 space 

space(n)
: Returns a string that consists of n spaces.
: Example of an SQL function usage fragment

> SELECT concat(space(2), '1')
: Result value

1





 spark_partition_id 

spark_partition_id()
: Returns the current partition ID.





 split 

split(str, regex)
: Splits str around occurrences that match regex.
The catalog list appears. Now you are connected to Presto engine in watsonx.data through Presto CLI.

presto:default> show catalogs;
Catalog
--------------
iceberg_data
jmx
system
tpcds
tpch
(5 rows)







 Connecting to Presto engine using Java/JDBC 



1. Download and install the [presto-jdbc-0.279.jar](https://prestodb.io/docs/current/installation/jdbc.html) on the client machine.
2. Add presto-jdbc-0.279.jar to the class path of your Java application.
3. Create a Java application by using JDBC interface. Following is an example for Java snippet:

public class PrestoJdbcSample
{
public static void main(String[] args) throws Exception
{
/
* example of fetching the location and credentials needed to connect, from
environment variables
/
String username = System.getenv("ENG_USERNAME");
String password = System.getenv("ENG_PASSWORD");
String hostname = System.getenv("ENG_HOST");
String portnumber = System.getenv("ENG_PORT");
String presto_url = "jdbc:presto://" + hostname + ":" + portnumber;
String ts_location = System.getenv("TRUSTSTORE");
// example: “ibm-lh-client/localstorage/volumes/infra/tls/truststore.jks";
String ts_password = System.getenv("TRUSTSTORE_PASSWORD");
// example: "changeit";
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
try
{
/ load the Presto JDBC Driver class /
If conssl.cfg does not exist or if any of the above parameters are not configured, the client keystore and stash file will default to $INFORMIXDIR/etc/client.kdb and $INFORMIXDIR/etc/client.sth. The IBM Informix client drivers are now ready to make SQLI connections using SSL.





* [Client side SSL setup needed for IBM Informix JDBC application](https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.jdbc.doc/ids_jdbc_490.htm)
* [Client side SSL setup needed for DRDA based non-Java client](https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.sec.doc/doc/t0053518.html?view=embed)
* [Client side SSL setup needed for DRDA based Java client (IBM JCC)](http://www.ibm.com/support/knowledgecenter/SSSNY3_10.1.0/com.ibm.db2.luw.apdv.java.doc/src/tpc/imjcc_c0024688.html)
* [Setting up MongoDB remote clients for SSL](http://www.ibm.com/support/knowledgecenter/SSB23S_1.1.0.13/gtpd5/tmdbssl.html)



For more information about configuring Informix client driver for SSL connection, refer to the Informix Security Guide's [Secure sockets layer protocol](http://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.sec.doc/ids_ssl_001.htm) topic in IBM Knowledge Center.





 InformixHQ 

The web-based monitoring and administration tool, InformixHQ, is also configured and running. The HQ Server and Agent processes run as the OS user "informixhqu".
Download the Presto executable jar from [https://prestodb.io/docs/current/installation/cli.html](https://prestodb.io/docs/current/installation/cli.html)
2. Rename the downloaded file as presto. Make it executable with chmod +x and run it.
3. To check whether Presto CLI is installed, run ./presto --version. Presto cli version displays. For example, Presto CLI 0.281-cfbc6eb
4. Run one of the following commands in the system where Presto CLI is installed.

<your-username> is optional if you have multiple connections with different users and want to differentiate them.



1. If you are using API key, run the following command.

./presto --server <https://Prestoengine host details> --catalog iceberg_data --schema default --username ibmlhapikey_<your-username> --password
2. If you are using IBM IAM token, run the following command.

./presto --server <https://Prestoengine host details> --catalog iceberg_data --schema default --username ibmlhtoken_<your-username> --password



5. Enter your IBM API key or IBM IAM token at the prompt.
6. At the Presto prompt, type show catalogs;. The catalog list appears. Now you are connected to Presto engine in watsonx.data through Presto CLI.

presto:default> show catalogs;
Catalog
--------------
iceberg_data
jmx
system
tpcds
tpch
(5 rows)







 Connecting to Presto engine using Java/JDBC 



1.
* The management IP ranges on either side might be overlapping with no loss in functionality.
* The cloud side has no vSphere visibility into the client side, which makes it more secure.







 HCX user portals 



* Client web user interface – The HCX client web portal is the main user interface for HCX. After the client-side HCX Manager is installed, it shows up as a snap-in to the vCenter web user interface. It controls remote cloud HCX registration (site pairing), fleet component deployment, network stretching, and VM migration into and out of the Cloud.
* Cloud side user interface – The cloud side HCX user interface is accessible through the public registration URL given for HCX client registration. By default, it uses the cloud side vCenter Admin credentials administrator@vsphere.local. It is typically used for upgrading the installation and modifying some network configuration.







 Related links 



* [Port access requirements for VMware HCX](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcx-archi-port-req)
* [Preparing the installation environment](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-planning-prep-install)
* [HCX Client deployment](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-vcs-client-deployment)
* [HCX on-premises Service Mesh](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-vcs-mesh-deployment)
* [VMware Hybrid Cloud migrations](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-migrations)
* [Monitoring parameters and components](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-monitoring)
HCX on-premises Service Mesh 

Review the following steps to configure the VMware HCX™ Client instance.



 Site pairing the VMware Solutions environment 



1. Log in to the VMware vSphere® Web Client.
2. From the Home menu, select the HCX option.
3. Under Infrastructure, InterConnect, click Add Site Pairing.



1. Set the Site URL to the HCX Cloud Manager URL, for example, https://x.x.x.x.x.
2. Set the username and password to the HCX Manager Admin Details: admin / password.

The previous details can be obtained from the IBM Cloud® for VMware Solutions console, under Services, HCX for the VMware vCenter Server® instance.



4. Click Connect.





 Results 

The Site Pairing is registered and displayed on the user interface.







 Creating HCX on-premises profiles 



 Creating on-premises Service Mesh network profiles 



1. Log in to the vSphere Web Client.
2. From the Home menu, select the HCX option.
3. Under Infrastructure, click InterConnect.
4. Under Multi-Site Service Mesh, click Network Profiles.
5. From Create Network Profile:



1. Select Distributed Port Group, for example, External.
2. Provide IP address ranges for External IPs, External Subnet Prefix Length, External Gateway, and DNS Details.
3. Set MTU to 1500.
4. Click Create.



6. Repeat the previous steps for Management and vMotion Networks. Set the MTU value to 9000.





 Results 



Table 1. Results

 Network Name MTU 

 External 1500 
 Management 9000 
 vMotion 9000 









 Creating on-premises Service Mesh compute profiles 



1. Log in to the vSphere Web Client.
2. From the Home menu, select the HCX option.
3. Under Infrastructure, click InterConnect.
4.
Use the ccli shell to check health and operation. After SSH is enabled and the L2C is connected, run the ip tunnel and ip link |grep t_ commands to view the status of the tunnels.





 Related links 



* [Glossary of HCX components and terms](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-components)
* [Preparing the installation environment](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-planning-prep-install)
* [HCX Client deployment](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-vcs-client-deployment)
* [HCX on-premises Service Mesh](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-vcs-mesh-deployment)
* [VMware Hybrid Cloud migrations](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-migrations)
* [Monitoring parameters and components](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-monitoring)
[2022-06-29 02:29:43,356] INFO Started o.e.j.s.ServletContextHandler@5562c2c9{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:915)
[2022-06-29 02:29:43,356] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:303)
[2022-06-29 02:29:43,356] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
5. Register the connectors.

The database and table must exist on Netezza Performance Server before you attempt to register the connectors.



* For the source connector, run the following command.

curl -s -X POST -H "Content-Type: Application/json" --data '{ "name": "test-source-jdbc", "config": { "connector.class": "io.aiven.connect.jdbc.JdbcSourceConnector", "tasks.max": "1", "connection.url":"jdbc:netezza://localhost:5480/db1;user=user1;password=secret","connection.user":"user1","connection.password":"secret","dialect.name":"GenericDatabaseDialect","topic.prefix":"my","mode": "bulk", "poll.interval.ms":"60000", "table.whitelist":"TEST_TABLE", "batch.max.rows":"10000000" } }' http://localhost:8083/connectors | jq
{
"name": "test-source-jdbc",
"config": {
"connector.class": "io.aiven.connect.jdbc.JdbcSourceConnector",
"tasks.max": "1",