vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Connecting to a Datastore 

To add and connect to a data store, complete the following steps in the Data Security Broker Manager:



1.  Login to Data Security Broker Manager.
2.  Select Databases from the left navigation and click Add Database + to add a data store.
3.  In the Add Database dialog, enter a name and description for the database in the Database Name and Database Description fields.
4.  Select the database type as Postgres from the Database Type drop-down list.
5.  Enter the IP address of the database in the Hostname or IP Address field.
6.  Specify the Port for the database and enter the user Database Username and Database Credential.
7.  Create a new user on your database for use with IBM Cloud Security Broker. For more information, see [Database Privileges](https://cloud.ibm.com/docs/security-broker?topic=security-broker-sb_db_priveleges) section.
8.  You can use only a Postgres database and enter your database name in the Postgres Database Name field.
9.  Optional: Select Use SSL, click Add file, and upload an SSL Certificate.
10. Click Add Database to complete enrolment. The new database appears in the list of configured databases.
11. Create the data that is required for encryption or decryption as tables in the new database that is created.
12. Once you have completed adding a database, the next step is to connect to a keystore. Refer to the [Add a Keystore in Data Security Broker Manager](https://cloud.ibm.com/docs/security-broker?topic=security-broker-sb_add_keystore) section to connect to a keystore in the Data Security Broker Manager.
Creating Terraform templates 

Learn how to create Terraform templates that are well-structured, reusable, and comprehensive.

A Terraform template consists of one or more Terraform configuration files that declare the state that you want to achieve for your IBM Cloud® resources. To successfully work with your resources, you must [configure IBM as your cloud provider](https://cloud.ibm.com/docs/schematics?topic=schematics-create-tf-configconfigure-provider) and [add resources to your Terraform configuration file](https://cloud.ibm.com/docs/schematics?topic=schematics-create-tf-configconfigure-resources). Optionally, you can use [input variables](https://cloud.ibm.com/docs/schematics?topic=schematics-create-tf-configconfigure-variables) to customize your resources.

You can write your Terraform configuration file by using HashiCorp Configuration Language (HCL) or JSON format.

Before you start creating your Terraform template, make sure to review the [IBM Cloud Schematics limitations](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-limitations).



 Configuring the provider block 

Specify the cloud provider that you want to use in the provider block of your Terraform configuration file. The provider block includes all the input variables that the IBM Cloud® Provider plug-in for Terraform requires to provision your resources.

Do I need to provide the IBM Cloud API key?

The IBM Cloud API key is essential to authenticate with the IBM Cloud platform. Also the IAM token and IAM refresh token that Schematics requires to work with the resource's API, and to determine the permissions that you were granted. When you use native Terraform, always you must provide the IBM Cloud API key. In Schematics, the IAM token is retrieved for all IAM-enabled resources, including IBM Cloud Kubernetes Service clusters, and VPC infrastructure resources.
If there is a dropdown, select a value from the dropdown.
* Click Done editing, if the editing is done.
* Click Save draft only if you need to save the draft to continue edit the input variables again later.





4. Click Create Blueprint that will redirect to the blueprint overview page.





 Verifying blueprint config creation using the UI 

Here are the steps to verify your blueprint config creation was successful.



1. Click your blueprint that is listed in the [Blueprints dashboard](https://cloud.ibm.com/schematics/blueprints) to view the results of the create operation.
2. Click Overview tab to see the blueprint summary, including Modules, Variables, Details. The Recent Job runs will show the summary details of the blueprint create job.
3. Click Modules tab to see the status of the resource modules. These will be in an Initialised state, indicating that the module workspaces have been created, but no resources created.
4. Click Jobs history tab view the result of the blueprint create job and the operations that are performed against the resource modules.
5. Click Settings tab to view the summary of the new blueprint configuration.



For more information, see the [troubleshooting section](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-create-fails).







 Creating a blueprint configuration using the API 

Follow the [steps](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-apics_api) to retrieve your IAM access token and authenticate with IBM Cloud Schematics by using the API. For more information, see [Create a blueprint config](https://cloud.ibm.com/apidocs/schematics/schematicscreate-blueprint) by using API.

The blueprint create API runs blueprint create, and blueprint jobsAPIs together, to run create blueprint operations.

Example

POST /v2/blueprints HTTP/1.1
Host: schematics.cloud.ibm.com
dests2[i] = retSchema.getFieldNameByIx (i + srcs1.length);
}

arg1Copier = retSchema.fieldCopier (arg1Schema, srcs1, dests1);
arg2Copier = retSchema.fieldCopier (arg2Schema, srcs2, dests2);
}

/
* Check the validity of tuple schemas given in the AQL “create function”.
* @see com.ibm.avatar.api.udf.TableUDFBasevalidateSchema(TupleSchema, TupleSchema, TupleSchema, Method, Boolean) for
* description of arguments
/
@Override
public void validateSchema (TupleSchema declaredInputSchema, TupleSchema runtimeInputSchema,
TupleSchema returnSchema, Method methodInfo, boolean compileTime) throws TableUDFException
{
// The output schema should contain the columns of the two input schemas in order.
AbstractTupleSchema arg1Schema = declaredInputSchema.getFieldTypeByIx (0).getRecordSchema ();
AbstractTupleSchema arg2Schema = declaredInputSchema.getFieldTypeByIx (1).getRecordSchema ();

System.err.printf ("TableConsumingTableFunc: Input schemas are %s and %sn", arg1Schema, arg2Schema);

// First check sizes
if (returnSchema.size () != arg1Schema.size () + arg2Schema.size ()) { throw new TableUDFException (
"Schema sizes don't match (%d + %d != %d)", arg1Schema.size (), arg2Schema.size (), returnSchema.size ()); }

// Then check types
Hudi X X X ✓ -- X X X X X ✓ X X X X X X X X 





 Limitations 



1. For CREATE TABLE, MySQL connector supports only CREATE TABLE AS.
2. For ALTER TABLE, MongoDB connector supports only table rename.
3. Db2 connector partially supports ALTER TABLE, CREATE VIEW, and DROP SCHEMA.
4. Netezza connector partially supports ALTER TABLE and CREATE VIEW.
5. MySQL, PostgreSQL, MongoDB, Db2, and Netezza connectors support DROP TABLE only when enabled in catalog.
6. The CREATE SCHEMA, CREATE TABLE, DROP SCHEMA, DROP TABLE, DELETE, DROP VIEW, ALTER TABLE, and ALTER SCHEMA are not available for database based catalogs in the Data Manager UI.
[Create a Schematics Actions](https://cloud.ibm.com/docs/schematics?topic=schematics-action-workingcreate-action).





 Referencing Ansible collections in your playbook 

Ansible collections group different reusable Ansible resources, such as playbooks, modules, and roles so that you can install them and use them in your playbook. Collections are stored in the [Ansible Galaxy](https://galaxy.ansible.com/) repository.

Similar to [Ansible roles](https://cloud.ibm.com/docs/schematics?topic=schematics-ansible-roles-galaxymain-file), collections require a specific folder structure in your GitHub repository.



1. Browse [Ansible Galaxy](https://galaxy.ansible.com/) to find the collection that you want to use in your playbook.
2. Create a requirements.yml file where you specify the collections that you want to install from Ansible Galaxy. For more information about how to structure this file, see the [Ansible documentation](https://docs.ansible.com/ansible/latest/galaxy/user_guide.htmlinstalling-collections). The following example uses the community.kubernetes collection.

collections:
- name: community.kubernetes
version: 0.9.0
3. Add a collections folder to your GitHub repository that is relative to your playbook, and store the requirements.yml file in this folder as shown in this example.

├── collections
└── requirements.yml
├── playbook.yaml
├── README.md
4. Reference a resource from your collection in your playbook. For more information, see the [Ansible documentation](https://docs.ansible.com/ansible/2.9/user_guide/collections_using.htmlusing-collections-in-a-playbook)







 Preparing your Ansible playbook to run in Schematics 

To prepare your Ansible playbook for Schematics, review the following considerations.
5. Click Apply plan to provision the configured resources.





 Verifying workspace plan 



1. Click your workspace that is listed in the [IBM Cloud console](https://cloud.ibm.com/schematics/workspaces) to view the results of the workspace details.









 Generate a workspace plan using the CLI 



1. Create a JSON file on your system and plan your workspace configuration. For more information about configuration options, see [ibmcloud schematics workspace new](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new) command.

Example

{
"name": "testwspace31jan",
"type": [
"terraform_v1.0"
],
"description": "terraform workspace",
"location": "us-east",
"tags": [
"department:HR",
"application:compensation",
"environment:production"
],
"template_repo": {
"url": "https://github.com/Anil-CM/newrepo"
},
"workspace_status": {
"frozen": true
},
"template_data": [{
"folder": ".",
"type": "terraform_v1.0",
"variablestore": {
"name": "sample_var",
"secure": true,
"value": "THIS IS IBM CLOUD TERRAFORM CLI DEMO",
"description": "Description of sample_var"
},
{
"name": "sleepy_time",
"value": "15"
}
]
}]
}

Syntax
You can create the effect of re-exporting the view or table in the current module by creating a new view:

select * from <imported_view_name_or_table_name>;
* The export view and output view statements that are shown in the examples are orthogonal to each other. In other words, an output view is not automatically an exported view, but it must be explicitly exported by using an export statement. An exported view is not automatically an output view, but it must be explicitly output by using the output view statement. In the examples, the export statement attempts to export the view PersonName.FirstName, which is an imported view. This attempt causes an error, which means that the developer must copy the imported view into a new view and then export that instead.







 Examples 

Example 1: Creating views and dictionaries and then exporting them for use in other modules

This example creates the views FirstName and NotFirstName. The view FirstName collects information about the given names that are represented in the FirstNamesDictionary dictionary. The other view collects the names that remain when you exclude the given names. Two dictionaries are necessary to make the text extraction easier. One dictionary contains all of the given names that you want to search for. The second dictionary, LastNamesDictionary, contains the surnames to search for. The FirstNamesDictionary dictionary is exported so that it can be used in other modules. The FirstName and the NotFirstName views are also exported so that they can be imported and used in other modules, such as module person in Example 2.

module personName;

create dictionary FirstNamesDictionary as
('Smith', 'John', 'Mary', 'Sam', 'George');

-- export dictionary statement

export dictionary FirstNamesDictionary;

create view FirstName as
extract dictionary 'FirstNamesDictionary'
on D.text as firstName
from Document D;

-- export view statement

export view FirstName;

create dictionary LastNamesDictionary as
('Stewart', 'Johnson', 'Smith', 'Hopkins', 'George');

create view NotFirstName as
The scripts create a Schematics workspace for you where you can see your script logs and the resources that are provisioned.

If you want to further customize your resources, you select Enable configuration customization after 'Create' and then Create. The scripts create the Schematics workspace for you and show you the parameters that you can customize. After you are done customizing the parameters, you select Generate Plan then Apply plan. The Terraform scripts run and create your resources.

For more information about Schematics, see [Getting started: IBM Cloud Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-getting-started).





 Next steps 

As the components are provisioned, you are directed to the Schematics workspace page for your deployment, where you can view the script logs.

After the Citrix DaaS for IBM Cloud components are provisioned, you need to complete a few post-provisioning steps to get you up and running. For more information, see [Post-provisioning steps for Citrix DaaS on VPC](https://cloud.ibm.com/docs/citrix-daas?topic=citrix-daas-post-provisioning-citrix-daas-vpc).
spark = SparkSession.builder .appName("lh-hms-cloud") .config("spark.hadoop.fs.s3a.bucket.lakehouse-bucket.endpoint" ,"s3.direct.us-south.cloud-object-storage.appdomain.cloud") .config("spark.hadoop.fs.s3a.bucket.lakehouse-bucket.access.key" ,"<access-key-for-source-bucket>") .config("spark.hadoop.fs.s3a.bucket.lakehouse-bucket.secret.key" ,"<secret-key-for-source-bucket>") .config("spark.hadoop.fs.s3a.bucket.source-bucket.endpoint" ,"s3.direct.us-south.cloud-object-storage.appdomain.cloud") .config("spark.hadoop.fs.s3a.bucket.source-bucket.access.key" ,"<access-key-for-lakehous-bucket>") .config("spark.hadoop.fs.s3a.bucket.source-bucket.secret.key" ,"<secret-key-for-lakehouse-bucket>") .enableHiveSupport() .getOrCreate()

return spark

def main():
try:
spark = init_spark()
 Create a database in lakehouse catalog
spark.sql("create database if not exists lakehouse.demodb LOCATION 's3a://lakehouse-bucket/'")
 list the database under lakehouse catalog
spark.sql("show databases from lakehouse").show()

 demonstration: Create a basic Iceberg table, insert some data and then query table
spark.sql("create table if not exists lakehouse.demodb.testTable(id INTEGER, name VARCHAR(10), age INTEGER, salary DECIMAL(10, 2)) using iceberg").show()
The following procedure describes how to create a LoopBack API by using the API Designer.



 Prerequisites 

Note: The following instructions assume that you are using the latest version of the developer toolkit. To check the latest version, see the [npm](https://www.npmjs.com/package/apiconnect) package page.

First, you need to create a LoopBack project using the CLI. Complete the following steps to do this:



1. To create a project, open a command line and enter:

apic loopback
2. In the prompt that follows, you are asked to name your application. Type a name then press Enter.

? What's the name of your application? (<application name>)

The tool prompts you to enter the name of the directory in which to create the project.

? Enter name of the directory to contain the project: (<project directory name>)
3. Press Enter to use the name you previously entered or type a new name then press Enter. The tool prompts you to select the kind of application to create:



? What kind of application do you have in mind? (Use arrow keys)
empty-server (An empty LoopBack API, without any configured models or datasources)
> hello-world (A project containing a basic working example, including a memory database)



4. Select the application that you want to use and press Enter. The tool displays a number of messages as it creates the project directory and adds a number of directories and files to it.It also runs npm install to install all the project dependencies, as specified in package.json. This process might take some time.
5. Before you can create a model you need to make sure that your current working directory is the project top-level directory. To do this, enter the following command:



cd <project directory name>

Note: Making your current working directory the project top-level directory will ensure that the option to add models and data sources is available when you open the API Designer.
SQL> COMMIT;
SQL> SELECT * FROM TDE_TS_TEST;
6. You can also create a table with an encrypted column and insert some test data:

SQL> CREATE USER Ctest IDENTIFIED BY test;
SQL> GRANT UNLIMITED TABLESPACE TO Ctest;
SQL> CREATE TABLE Ctest.tde_test (id NUMBER(10), data VARCHAR2(50) ENCRYPT);
SQL> INSERT INTO Ctest.tde_test VALUES (1, 'This is also a secret!');
SQL> COMMIT;
SQL> SELECT * FROM Ctest.tde_test;

 Verify encrypted tablespace and encrypted column
SQL> SELECT TABLESPACE_NAME, ENCRYPTED FROM DBA_TABLESPACES;
SQL> SELECT * FROM dba_encrypted_columns ;







 Next steps 

Your sensitive data is now stored safely in encrypted tables paces and encrypted columns. And the TDE master encryption key is kept in Hyper Protect Crypto Services in a highly secure and tamper-proof manner.

In this tutorial, you learned how to set up Oracle Database TDE with Hyper Protect Crypto Services.



* Learn more about

[PKCS #11](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-pkcs11-intro).
* Learn more about the

[PKCS #11 API](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-pkcs11-api-ref).

[Start to use the PKCS #11 API](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-set-up-pkcs-api).
After you created an external data source, you can create an external table that accesses the yellow taxi data from January 2022.

Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);

The DATAOBJECT argument must reference a single file in the parquet format. If you want to query from multiple parquet files, you must create additional external tables.

Example:

create EXTERNAL table YELLOW_TAXI_JANUARY_2022 on EXAMPLEDATASOURCE
using ( 
 DATAOBJECT ('/yellow_tripdata_2022-01.parquet') FORMAT 'PARQUET' 
);





 3. Load the data into Netezza Performance Server. 

To load data from the data lake into a Netezza Performance Server table, run CREATE TABLE AS SELECT from the external table that you want to load.

create table 'TABLE NAME LOADED' as
select
     
from
     'TABLE';

Example:

create table YELLOW_TAXI_JANUARY_2022_LOADED as
select
     
from
        YELLOW_TAXI_JANUARY_2022;

INSERT 0 2463931





 4. Query the loaded data. 

Now, you can query the loaded data by using the improved read/write performance, zonemaps, etc.

The parquet column names are case sensitive. You must use double quotation marks ("") when you are querying specific columns.



* To identify the passenger count, run:

select
sum("passenger_count")
from YELLOW_TAXI_JANUARY_2022_LOADED;

Output:

SUM
-------
3324167
(1 row)

You do not have to load whole tables into Netezza Performance Server.
Related workspace: name=<prefix>-003-cos-module, sourcerelease=(not specified), sourceurl=https://github.com/Cloud-Schematics/cos-module, folder=.
--- Ready to execute the command on Agent jobrunner-89b7f499-slhf9 ---
.....



* Now, you are ready to use the provisioned COS service to store your objects or files.
* You can also deploy your Schematics Workspaces template, and bind your Agent to provision, install the software, or a tool in your private network to the IBM Cloud Schematics service.
* You can also configure, and operate on your cloud cluster resources without any time, network, or software restrictions.







 Next steps 

You have completed the entire Schematics Agent set up and working flow.



* Looking for Agents samples? Check out the [IBM Cloud Schematics GitHub repository](https://github.com/Cloud-Schematics?q=Agent&type=all&language=&sort=).
* For any challenges in Agents set up, see [FAQ about Agent](https://cloud.ibm.com/docs/schematics?topic=schematics-faqs-agent) and [Troubleshooting guide](https://cloud.ibm.com/docs/schematics?topic=schematics-agent-crn-not-found).
* Make sure that you have the [permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-access) to create a workspace.



Ensure the location and the url endpoint are pointing to the same region when you create or update the Schematics Workspaces and actions. For more information about location and endpoint, see [Where is your information stored](https://cloud.ibm.com/docs/schematics?topic=schematics-secure-datapi-location)?





 Creating a workspace using the UI 



1. Log in to [IBM Cloud](https://cloud.ibm.com/).
2. Access Schematics > Workspaces > [Create workspace](https://cloud.ibm.com/schematics/workspaces/create).



* In Specify Template section:



* GitHub, GitLab, or Bitbucket repository URL - <provide your Terraform template Git repository URL.
* Personal access token - <leave it blank>.
* Terraform Version - terraform_v1.0. You need to select Terraform version 1.0 or greater version. For example, if your Terraform templates are created by using Terraform v1.0, select the Terraform version parameter as terraform_v1.0. You can select Terraform_v1.1 to use Terraform version 1.1, terraform_v1.0 to use Terraform version 1.0. When you specify terraform_v1.1means that users can have template that is of Terraform v1.1.0, v1.1.1, or v1.1.2, so on. Schematics supports Terraform_v1.x and also plans to make releases available after 30 to 45 days of HashiCorp Configuration Language (HCL) release.

Schematics supports the current release of Terraform v1.1, through terraform_v1.0. The Terraform template must use the version constraint, such as > or >= or > for the required_version of Terraform, to automatically pick the current version.
ibmcloud schematics blueprint create 

Create a blueprint config from the command line using the ibmcloud schematics blueprint create command. Creating a blueprint config is the first step to deploy a blueprint environment. The blueprint config is created from user provided parameters via the command line that specifies the source of the blueprint template in a Git repository, and input values sourced from Git hosted input files and locally sourced input values.

The create command supports passing all required parameters through CLI and passing input values from local files.

A experimental file option is provided to create a blueprint config from a configuration provided as a JSON file from the local file system. Refer to the [ibmcloud schematics blueprint create - file option](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-blueprint-create) section. These two usage modes are mutually exclusive.

For Schematics Blueprints, the [Schematics plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cliinstall-schematics-plugin) version must be greater than the 1.12.5 version.

Refer to the section on [Creating Blueprints](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-blueprint-create) for examples of command syntax and output.

Before your begin:



* Install or update the [Schematics plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cliinstall-schematics-plugin) version that is greater than the 1.12.5.
* Select the IBM Cloud region that you wish to use to manage your Schematics. Set the region by running [ibmcloud target -r <region>](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_cliibmcloud_target) command.