vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Whenever your document structure changes, you need to reexecute the FLUSH SAMPLE command to re-create the BI Connector schema.

If your database has many documents, schema creation is a time-consuming process. You should only run FLUSH SAMPLE if your document structure changes or you create new collections.





 Step 7: Visualize your data! 

You can now select the WHO database on the right. Next, press "Update Now" to see the data in tabular format.

Zoom

![Visualize your data in tabular format](https://cloud.ibm.com/docs-content/v1/content/5561b38c6df9e8168523d143eaf64ab6acd7b600/cloud-databases/images/tableau-connect-view-data.png)

Figure 5. Visualize your data in tabular format

If you select the map view from the right menu, you can also see your data in a nice map, with the columns containing country codes and the rows containing cumulative cases.

Zoom

![Visualize your data in map view](https://cloud.ibm.com/docs-content/v1/content/5561b38c6df9e8168523d143eaf64ab6acd7b600/cloud-databases/images/tableau-connect-map-view.png)

Figure 6. Visualize your data in map view





 Step 8: Deprovision your infrastructure 

To avoid incurring charges, remember to deprovision your resources. Terraform can take care of this. In the terminal, make sure you are in the project's terraform directory, then use the following command:

terraform destroy --auto-approve

To confirm that your instance is deprovisioned, check the Resources section in the [IBM Cloud Console](https://cloud.ibm.com/resources).







 Summary 

In this tutorial, you used Terraform to provision a Databases for MongoDB EE (Enterprise Edition) instance and then used the Analytics Add-On BI Connector to visualize your data in Tableau.
and documentCreationDate Text;

Example 2: Require document statement with varying column types

The following sample defines a document schema that contains columns of varying field types.

module sample;

-- Require document statement with varying field types

require document with columns
bookText Text
and purchaseCount Integer
and bookPrice Float
and isBookPopular Boolean;

Example 3: Merging document schemas

The example describes how to merge document schemas that use the files first.aql, last.aql, and socialsecurity.aql.

first.aql:
module person;
require document with columns firstName Text;

last.aql:
module person;
require document with columns lastName Text;

socialsecurity.aql
module person;
require document with columns lastName Text
and socialSecurityNo Integer;

The merged schema is (firstName Text, lastName Text, socialSecurityNo Integer).







 The create view statement 

The top-level components of an AQL extractor are its views. Views are logical statements that define, but do not necessarily compute, a set of tuples.



 Syntax 

The create view statement can take one of three forms. The simplest form defines a logical view that is composed of the tuples of a single select or extract statement. The second is a multijoin form that defines a view that comprises the tuples that arise from the multiset union of severalselect or extract statements. The third form defines a new view that contains the set difference between tuples from two select or extract statements.

create view <viewname> as <select or extract statement>;

create view <viewname> as (<select or extract statement>) union all (<select or extract statement>)...;

create view <viewname> as (<select or extract statement>) minus (<select or extract statement>);





 Description 



* <viewname>

The <viewname> can be a simple identifier or a double-quoted identifier. It cannot contain the period character.
* <select or extract statement>
Alternatively, you can also register and attach an object storage bucket with pre-existing data to HMS.

ORC, Parquet, Avro, RCFile, SequenceFile, JSON, Text (CSV) are the Hive supported file formats.







 Step 2: Load data files into Presto 

After attaching the object storage bucket to HMS, you need to load data files into Presto by creating schema and external tables through the Hive connector.



1. Run the following command to create schema for the data you want to access.

CREATE SCHEMA <SCHEMA_NAME> WITH ( location = '<SCHEMA_LOCATION>' );

For example:

CREATE SCHEMA hive.gosales WITH ( location = 's3a://lhbeta/gosales' );
2. Run the following command to create table by using an external location by pointing to an existing table.

CREATE TABLE IF NOT EXISTS <TABLE_NAME> ("<COLUMN_NAMES>" <DATA_TYPE>) WITH ( format = '<DATA_FORMAT>', external_location = '<TABLE_LOCATION>' );

For example:

CREATE TABLE IF NOT EXISTS hive.gosales.branch ("BRANCH_CODE" int, "ADDRESS1" varchar, "ADDRESS1_MB" varchar, "ADDRESS2" varchar, "ADDRESS2_MB" varchar, "CITY" varchar, "CITY_MB" varchar, "PROV_STATE" varchar, "PROV_STATE_MB" varchar, "POSTAL_ZONE" varchar, "COUNTRY_CODE" int, "ORGANIZATION_CODE" varchar, "WAREHOUSE_BRANCH_CODE" int) WITH ( format = 'CSV', external_location = 's3a://lhbeta/gosales/branch' );







 Step 3: Generate statistics with analyze table 

If you want to use the data without creating a new copy for a different table format or more table optimizations, you can generate statistics alone with analyze table.



1. To generate statistics with analyze table, run the following command:

analyze <TABLE_NAME>;

For example:
Synonyms 



  Creating synonyms 



1.  Go to Databases
2.  Select the database in which you want to create a synonym.
3.  Select the schema in which you want to create a synonym.
4.  Go to DB Objects > Synonyms.
5.  Click Create synonym.
6.  Type a name for the synonym. The name can be up to 128 characters long and it must begin with a letter or underscore and cannot contain embedded spaces. The name must be unique.
If the name contains special characters, enclose it in double quotation marks.
7.  Specify the reference type.
For example, table or view.
8.  Specify the name of the reference type for which you are creating this synonym.
9.  Click Create.
About Data manager 

The Data manager page in IBM® watsonx.data is the entry point to browse the schemas and tables by engine. You can select an engine to view the associated catalogs, schemas, and tables.

From the Data manager page, you can create schemas and tables by using the Create option that is provided in the left window. You can also select a catalog or schema, click the overflow menu, and use the corresponding Create option to create a schema or table. Create table from file option in the overflow menu of schema is also used to ingest a data file into watsonx.data. Similarly, schemas and tables can be dropped from the catalogs.

Wait for a few minutes to view the changes after a schema or table is dropped.

Adding, renaming, or dropping a column are the other tasks that can be performed in the Data manager page.

You can browse the Table schema and upto 25 rows of Data sample for some tables. You can view the Time travel snapshots and use the Rollback feature to rollback to a given snapshot for Iceberg tables.

Presto cannot roll back to the snapshots that are not ancestors of the current snapshot.
contactName string,
contactTitle string,
address string,
city string,
region string,
postalCode string,
country string,
phone string,
fax string
)
USING CSV
PARTITIONED BY (COUNTRY)
location cos://us-geo/sql/customers_partitioned.csv

-- attach table partitions by scanning the location of the table
ALTER TABLE customers_partitioned RECOVER PARTITIONS
Show more

An alternative way to create a table definition is to use the automatic schema detection where you do not need to specify any columns.

-- create a definition for the table shippers with automatic schema detection
CREATE TABLE shippers
USING parquet
location cos://us-geo/sql/shippers.parquet





 tableProperty 

The tableProperty option consists of one or more key and value pairs.



Table 56. tableProperty option.

 Key Value Default Description 

 HEADER true or false true Use the HEADER option to specify whether your CSV object has a header included. 
 DELIMITER single (possibly escaped) character , (comma) Use the DELIMITER option to specify the used delimiter in your CSV objects. All single Unicode characters are allowed as delimiters. 
 MULTILINE true or false false Use the MULITLINE option to specify whether the JSON object is single or multiline. 



-- Example of creating a table definition in the catalog for a CSV data without header line:
CREATE TABLE LAX_CARGO
(Extract_Date timestamp, Report_Date timestamp, Arrival_Departure string,
Domestic_International string, Cargo_Type string, Air_Cargo_Tons int)
USING csv
location cos://us-geo/sql/LAX_Cargo.csv
options(HEADER=false)







 Drop Table 



 dropTable 

Drop a table definition from the catalog. If the table does not exist, you receive an error, unless the IF EXISTS option is specified.

This command does not delete any data in Object Storage. It removes only the table definition from the catalog.

-- drop a definition for the table customer
Libraries 



  Creating libraries 



1.  Go to Databases
2.  Select the database in which you want to create a view.
3.  Go to DB Objects > Libraries.
4.  Click Create library.
5.  Select the schema where you want to create a library.
6.  Type a name for the library.
7.  Specify the external host object filename.

This is the path for the compiled host object file of the shared library. For example, /etc/motd.
8.  Specify the external SPU object filename.

This it the path for the compiled object file of the Linux SPU. For example, /etc/motd.
9.  Click Create.



You can also select to replace your library, include a load clause, and include a dependencies clause.
So, if you are either familiar with the schema, or want to repetitively use the data for queries, create a table in the catalog. Such a table improves performance for repeated query executions.

Another advantage of creating a table in the catalog is that the table name serves as an alias and is decoupled from the data location. Hence, you can separate the tasks of data engineers and SQL authors. Data engineers deal with the data location and publish registered tables in the catalog by using descriptive table names. Hence, SQL authors are able to compose queries without having to know the exact location and format of data on Object Storage. If the data location changes, only the table in the catalog must be updated, but the table name remains unchanged. Updates of the physical data structure are simplified and the robustness of SQL statements and applications is increased.





 Usage 

You manage the database catalog in Data Engine with Database Definition Language (DDL) statements that you submit just like any other SQL query statement. The catalog is stored independently of Object Storage: No data is written to Object Storage when you create or change table definitions, and no data is deleted from Object Storage when you drop a table definition. To call the catalog management statements, you need the Manager user role assigned.

To register a new table in the catalog, use the CREATE TABLE statement, as in the following example:

CREATE TABLE employees
USING PARQUET
LOCATION cos://us-geo/sql/employees.parquet

The statement automatically detects the schema of the data at the location that is indicated. See the [SQL reference](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencecreateTable) for options that can be set on the table.

Use the DESCRIBE TABLE statement to verify the detected table schema:

DESCRIBE TABLE employees

If the DESCRIBE TABLE output shows partition information, you must run an ALTER TABLE ... RECOVER PARTITIONS statement to attach the partitions. For more information, see the section on [partitioned tables](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting_started_catalogpartitioned).
* [Defining a JSON config file](https://cloud.ibm.com/docs/schematics?topic=schematics-create-blueprint-filebp-create-configJSON)



[Using environment variables with blueprints](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-env-varsbp-env-vars)



* [Blueprints usage](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-env-varsusage)







 Managing Workspaces 



 Creating workspace 

[Creating workspaces and importing your Terraform template](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wkssch-create-wks)



* [Before you begin](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wksprerequisites-create)
* [Creating a workspace using the UI](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wkscreate-wks-ui)



* [Importing your Terraform template](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wksimport-template)
* [Using Terraform templates in IBM Cloud](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wksrun-template)



* [Creating a workspace using the CLI](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wkscreate-wks-cli)
* [Creating a workspace using the API](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wkscreate-wks-api)
Sequences 



  Creating sequences 



1.  Go to Databases
2.  Select the database in which you want to create a sequence.
3.  Select the schema in which you want to create a sequence.
4.  Go to DB Objects > Sequences.
5.  Click Create sequence.
6.  Type a name for sequence. The name can be up to 128 characters long and it must begin with a letter or underscore and cannot contain embedded spaces. The name must be unique.
If the name contains special characters, enclose it in double quotation marks.
7.  Specify the starting integer value for the sequence.
8.  Specify the minimum value.
9.  Specify the maximum value.
10. Specify the increment by value.
11. Specify whether you want the sequence to restart when it reaches its last value by checking the Cycle box.
The default is No cycle, which means that the sequence stops when it reaches its last value.
12. Click Create.
For more information, see [Infrastructure deployment with Schematics Workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-how-it-workshow-to-workspaces).
2. Create your Ansible playbook. Use one of the [IBM-provided playbooks](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplates) to get started or browse [Ansible Galaxy](https://galaxy.ansible.com/) to find existing roles and collections. You can then reference these [roles](https://cloud.ibm.com/docs/schematics?topic=schematics-ansible-roles-galaxymain-file) and [collections](https://cloud.ibm.com/docs/schematics?topic=schematics-create-playbookschematics-collections) in your playbook.
3. Create a repository in GitHub or GitLab, and build the Ansible playbook directory and file structure. Depending on whether you use Ansible roles and collections, this directory structure might vary. To find a sample structure, see this [sample playbook](https://github.com/Cloud-Schematics/ansible-app-deploy-iks/blob/master/site.yml).
4. Upload your Ansible playbook, modules, roles, and collections to your GitHub repository.
5. [Create a Schematics Actions](https://cloud.ibm.com/docs/schematics?topic=schematics-action-workingcreate-action).





 Referencing Ansible collections in your playbook 

Ansible collections group different reusable Ansible resources, such as playbooks, modules, and roles so that you can install them and use them in your playbook. Collections are stored in the [Ansible Galaxy](https://galaxy.ansible.com/) repository.
Procedures 



  Creating procedures 



1.  Go to Databases.
2.  Select the database in which you want to create a procedure.
3.  Select the schema in which you want to create a procedure.
4.  Go to DB objects > Procedures.
5.  Click Create procedure.
6.  Type a name for the procedure. The name can be up to 128 characters long and it must begin with a letter or underscore and cannot contain embedded spaces. The name must be unique.
If the name contains special characters, enclose it in double quotation marks.
7.  Specify the arguments.
This is to specify the parameter type list of the procedure. For example, int.
8.  Specify the return type of your procedure.
For example, int.
9.  Specify the procedure body.
This is the source code for the body of the stored procedure. For example

BEGIN RAISE NOTICE 'The customer name is David'; END;
10. Specify whether to replace the procedure.
11. Specify whether to include an execute clause.
12. Click Create.
Creating an Ansible playbook 

Follow these [prerequisites](https://cloud.ibm.com/docs/schematics?topic=schematics-create-playbookplan-ansible-playbook) and general steps to create your Ansible playbook. For detailed information about how to structure your playbook, see the [Ansible documentation](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html) or this [blog](https://www.ansible.com/blog/getting-started-writing-your-first-playbook).

Want to use existing Ansible playbooks to get started? Try out one of the [IBM-provided Ansible playbooks](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplates) or browse existing Ansible collections and roles in [Ansible Galaxy](https://galaxy.ansible.com/)



1. [Create your resource inventory where you want to run your Ansible playbook](https://cloud.ibm.com/docs/schematics?topic=schematics-inventories-setup). You can also use the built-in Terraform capabilities in Schematics to provision your target hosts. For more information, see [Infrastructure deployment with Schematics Workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-how-it-workshow-to-workspaces).
2. Create your Ansible playbook. Use one of the [IBM-provided playbooks](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplates) to get started or browse [Ansible Galaxy](https://galaxy.ansible.com/) to find existing roles and collections.
spark.sql("create database if not exists lakehouse.demodb LOCATION 's3a://lakehouse-bucket/'")
 list the database under lakehouse catalog
spark.sql("show databases from lakehouse").show()

 demonstration: Create a basic Iceberg table, insert some data and then query table
spark.sql("create table if not exists lakehouse.demodb.testTable(id INTEGER, name VARCHAR(10), age INTEGER, salary DECIMAL(10, 2)) using iceberg").show()
spark.sql("insert into lakehouse.demodb.testTable values(1,'Alan',23,3400.00),(2,'Ben',30,5500.00),(3,'Chen',35,6500.00)")
spark.sql("select * from lakehouse.demodb.testTable").show()

 demonstration: Ingest parquet and csv data into a wastonx.data Iceberg table
 load parquet data into dataframce
df = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-01.parquet")
 write the dataframe into an Iceberg table
df.writeTo("lakehouse.demodb.yellow_taxi_2022").create()
 describe the table created
spark.sql('describe table lakehouse.demodb.yellow_taxi_2022').show(25)
 query the table
spark.sql('select * from lakehouse.demodb.yellow_taxi_2022').count()

 load csv data into a dataframe
csvDF = spark.read.option("header",True).csv("s3a://source-bucket/zipcodes.csv")
csvDF.createOrReplaceTempView("tempCSVTable")
 load temporary table into an Iceberg table
For detailed syntax and privileges, see [the CREATE DATABASE command](https://www.ibm.com/docs/en/netezza?topic=npsscr-create-database-2).

CREATE DATABASE <db_name> DATA_VERSION_RETENTION_TIME <number-of-days>;

Example:

CREATE DATABASE DB1 DATA_VERSION_RETENTION_TIME 30;







 Creating time travel objects with the web console 



 Creating temporal tables with the web console 



1. Log in to the web console as described in [Getting started with the web console](https://cloud.ibm.com/docs/netezza?topic=netezza-getstarted-console).
2. Create a temporal table as described in [Creating tables](https://cloud.ibm.com/docs/netezza?topic=netezza-create-tablescreating-tables).



You must set retention time interval to a nonzero value.

Databases, schemas, and table names containing a dot character (".") do not show in the time travel statistics and graphs when you set the retention time interval to a nonzero value. When you do not set the retention time interval, all special characters are supported.

When you insert a row into the table, the row receives a virtual insert timestamp that is equal to the commit time of the inserting transaction.

When you delete a row from the table, the row receives a virtual delete timestamp that is equal to the commit time of the deleting (or truncating) transaction.





 Creating temporal schemas with the web console 

To create a temporal schema, set retention time interval to a nonzero value.



1. Log in to the web console as described in [Getting started with the web console](https://cloud.ibm.com/docs/netezza?topic=netezza-getstarted-console).
2. Create a temporal schema as described in [Creating schemas](https://cloud.ibm.com/docs/netezza?topic=netezza-create-schemas).
From the [Schematics Workspaces dashboard](https://cloud.ibm.com/schematics/workspaces), click Create workspace.
2. In Specify template section. Enter GitHub, GitLab, or Bitbucket Repository URL as

https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-resource-instance
3. Select terraform_v01.0 from the Terraform version drop down.
4. Click Next.
5. In the Workspace details section, enter your Workspace name, Tags, Resource group, Location, and Description.

Ensure you provide the right resource group, and the location where you want to create the workspace.
6. Click Next and then click Create to create Schematics Workspaces successfully.
7. In the Variables section, template variables are displayed. Optionally, override the variables by referring the [readme file](https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-resource-instance).

If you already have an existing IBM Cloud Object Storage instance in your account, you must enter standard in the plan field.
8. Click Generate plan to see the plan is generated successfully.
9. Click Apply plan. This process might take a few minutes to complete. Click Jobs to see the details of the provisioning process.





 Output 

View the provisioned IBM Cloud Object Storage instance.



1. From the [IBM Cloud resource list](https://cloud.ibm.com/resources), select the Storage to view the provisioned IBM Cloud Object Storage instance.
2. For more information, to create IBM Cloud Object Storage bucket, see [create some buckets to store your data](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storagegs-create-buckets).
def generate_and_store_data(spark,sc):
    data =[("India","New Delhi"),("France","Paris"),("Lithuania","Vilnius"),("Sweden","Stockholm"),("Switzerland","Bern")]
    columns=["Country","Capital"]
    df=spark.createDataFrame(data,columns)
    df.write.mode("overwrite").parquet("cos://mybucket.mycosservice/countriescapitals.parquet")

def main():
    spark,sc = init_spark()
    generate_and_store_data(spark,sc)
if __name__ == '__main__':
    main()

Show more





 Creating schema 

Create the metastore table schema definition in the data engine. Note that you can't use standard Spark SQL syntax to create tables when using Data Engine as a metastore. There are two ways that you can use to create a table:

- From the Data Engine user interface or by using, the standard Data Engine API (see [Data Engine service REST V3 API](https://cloud.ibm.com/apidocs/sql-query-v3introduction)) or Python SDK (see [ibmcloudsql](https://pypi.org/project/ibmcloudsql/)).

Example
: Enter:

 {: codeblock}
CREATE TABLE COUNTRIESCAPITALS (Country string,Capital string) 
USING PARQUET 
LOCATION cos://ALIAS NAME/mybucket/countriescapitals.parquet

Parameter values:
ALIAS NAME: The data engine endpoint for your region. For more information on the currently supported data engine endpoints, see [Data engine endpoints](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewendpoints). Make sure that you select the standard aliases.