vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* When multiple modules are combined to form an extractor, the schema of the Document view of the entire extractor is defined by the duplicate-free union of Document schemas for each module. An exception is raised when any column found across the multiple require document with columns statements is found to be conflicting in its type requirements across modules. For example, a module requires a column X with type Y when another module that is being loaded with it requires a column X with type Z.
* An exception is raised when you run an extractor if the provided input document tuple does not contain all of the required columns. An exception is also raised if a column does not conform to its corresponding required type.
* When the require document with columns statement is present inside a module, every column of the special view Document that is referenced must be declared in at least one of the require document with columns statements. The statement can be found in different AQL files within the same module. However, all such require document with columns statements would be merged at the level of the module to form a module-wide require document with columns statement.







 Examples 

Example 1: Require document statement with similar column types

The following example defines a document schema that contains four fields of the same type. This AQL sample expects each document tuple of the data collection to contain four columns as defined in the document schema.

Refer to JSON document formats for details on how to create a document that conforms to a schema.

module person;

-- Require document statement with similar field types

require document with columns
inputDocumentText Text
and inputDocumentLabel Text
and documentAuthor Text
and documentCreationDate Text;

Example 2: Require document statement with varying column types

The following sample defines a document schema that contains columns of varying field types.

module sample;

-- Require document statement with varying field types

require document with columns
bookText Text
and purchaseCount Integer
and bookPrice Float
and isBookPopular Boolean;

Example 3: Merging document schemas
Ideally, create tables by using the [Cloud Object Storage catalog](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-hivemetastore). Data skipping complements these best practices and provides significant cost savings and performance benefits.

To use this feature, you must create indexes on one or more columns of the data set. Start by indexing columns that you query most often in the WHERE clause. A full stop (".") in any column name is not supported.

The following four index types are supported:



 Index type Description Applicable to predicates in WHERE clause Column types 

 MinMax Stores minimum or maximum values for a column <,<=,=,>=,> and geospatial functions where applicable All types, except for complex types. [Supported Spark SQL data types](https://spark.apache.org/docs/latest/sql-reference.htmldata-types) 
 ValueList Stores the list of unique values for the column =,IN,LIKE All types, except for complex types. [Supported Spark SQL data types](https://spark.apache.org/docs/latest/sql-reference.htmldata-types) 
 BloomFilter Using bloom filter technique for set membership =,IN Byte, string, long, integer, short 
 Geospatial Stores a geospatial bounding box Geospatial functions [Geometry types](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_datatypes.html) 



Use ValueList for a column if the number of distinct values for that column per object is typically much smaller than the total number of values for that column per object (otherwise, the index can be undesirably large). Use BloomFilter if the number of distinct column values per object is high.

Indexes, or data skipping metadata, are stored in a location you specify. Metadata is typically much smaller than the data itself. If changes are made to some of the objects in the data set after index creation, refresh the indexes.
+ arg1Schema.size (), returnSchema.getFieldTypeByIx (i + arg1Schema.size ()), arg2Schema.getFieldTypeByIx (i)); }
}
}

}
Show more







 Declaring user-defined functions 

You can make the user-defined scalar functions and machine learning models from PMML files available to AQL by using the create function statement.



 Syntax 

The general syntax of the create function statement is as follows:

create function <function-name>(<input-schema-definition>)
return <return-type> [like <column-name>] | table ( <output-schema-definition)
external_name <ext-name>
language [java | pmml]
[deterministic | not deterministic]
[return null on null input | called on null input];

<input-schema-definition>
<column-name> <data-type> | table (<output-schema-definition>) as locator [,<column-name> <data-type> | table (<output-schema-definition>) as locator ]

<output-schema-definition>
<column-name> <data-type> [,<column-name> <data-type>]





 Description 



* <function-name>

The <function-name> declares the AQL name of the UDF. The UDF is referred to in the AQL code with this name
* <input-schema-definition>

Specifies the input parameters of the UDF. An input parameter has a name, which is specified as <column-name>, and can be either a scalar type or a table locator. When the language is PMML, the function must take a single table that is called params as the argument.
* <column-name>

Specifies the name of a column in the input or the output of the UDF.
* <data-type>
Follow the Blueprints YAML language extension and blueprint schema [installation and configuration instructions](https://github.com/Cloud-Schematics/vscode-blueprint-schema).

Schematics Blueprints works with definitions sourced from a version control system. All input files, templates, and modules are all maintained in online Git source control. Optionally [configure VScode](https://code.visualstudio.com/docs/sourcecontrol/overview) for your Git environment.





 Sourcing a blueprint template 

Initially you can start by customizing the [sample template](https://github.com/Cloud-Schematics/blueprint-basic-example) or use one of the deployable template examples in the [IBM Cloud Schematics GitHub repository](https://github.com/orgs/Cloud-Schematics/repositories/?q=topic:blueprint).





 Clone source template repo in Git 

Create a copy of the source template in your own Git repository or version control system. These instructions refer to use with Github where both the template and input file are stored in the same repo. Separate repositories can be used to version the template and inputs independently.

Alternatively work with your own template library in Github or Gitlab. These can be either public or private repos.

Instructions to clone one of the [IBM Cloud Schematics GitHub repository](https://github.com/orgs/Cloud-Schematics/repositories/?q=topic:blueprint) examples:



* Navigate to the main page of the source template repository
* Click Use this template/repository
* Select Create a new repository
* Use the Owner drop-down menu, and select the account where want to create the new repository
* Type a name for your repository
* Click Create repository from template







 Clone repo to local machine 

Clone your new template repository to your local machine for editing:



* Navigate to the main page of your new repository
* Click <> Code (Download Code)
new partitions can add data for that column
ALTER TABLE customers_addcol ADD COLUMNS (priority INTEGER)

Do not use the ADD COLUMNS option with CSV tables. The CSV data format identifies columns by order (not by name), so any schema change leads to a schema mismatch with existing data.

Alternatively, you can perform schema changes by dropping and re-creating catalog tables. It does not affect the stored data in Object Storage. This allows you to reexecute the automatic schema detection when the underlying data is extended with new objects containing more columns. You can also use this method to remove columns from the schema that you do not want to appear in the catalog.







 Describe table 



 describeTable 

Return the schema (column names and data types) of a table or view definition. If the table or view does not exist, you receive an error.

-- returns detailed information about the customer table
DESCRIBE TABLE customers_partitioned







 Show tables 



 showTables 

Returns the list of the defined tables and views in the catalog. The LIKE option allows to filter for an indicated pattern. Use * as wildcard character.

-- returns all defined tables in the catalog for this instance
SHOW TABLES







 Show Partitions 



 showPartitions 

List the defined partitions of a table when a table was created as partitioned. You can filter the returned partitions by using the partitionSpec option.

-- returns all partitions for the table customers_partitioned
SHOW PARTITIONS customers_partitioned









 Index management 

With the following commands, you can create indexes for data skipping during SQL execution to improve performance and lower the costs of your SQL queries. The indexes store summary metadata for each partition of your table to avoid scanning data that is not needed for the query execution. For more information, see [index management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-index_management).



 Create index 



 createIndex 

Create an index on the objects in the specified Object Storage location or on the specified table. Define the required index type for each column that you want to calculate the summary metadata for.
Using Workspaces to deploy infrastructure and cloud services 

Use one of the IBM provided templates to create an [IBM Cloud® Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cloud-object-storage) service instance that you can use to store your data in IBM Cloud.

The IBM Cloud Schematics template is a set of files that define the IBM Cloud resources that you want to create, update, or delete. You create a Schematics Workspaces that points to your template and use the built-in capabilities of the IBM Cloud provider plug-in for Terraform to provision your IBM Cloud resources. For more information about the provider and how Schematics spins up your IBM Cloud resources, see [Infrastructure deployment with Schematics Workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-how-it-workshow-to-workspaces).



 Before you begin 

Before you can use this template, you must complete the following tasks.



* Make sure that you have the permissions to [create a Schematics Workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-accessaccess-roles).
* Make sure that you have the permissions to [create an IBM Cloud Object Storage instance](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam).







 Creating an IBM Cloud Object Storage instance with Schematics 

Use the IBM provided Terraform template to provision an IBM Cloud Object Storage instance with a Schematics Workspace.



1. From the [Schematics Workspaces dashboard](https://cloud.ibm.com/schematics/workspaces), click Create workspace.
2. In Specify template section. Enter GitHub, GitLab, or Bitbucket Repository URL as

https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-resource-instance
3.
2. Click Assets > Dictionaries.
3. Click Create Dictionary to add a dictionary.

Do not click Upload dictionary, which is used to upload a dictionary that you want to use as-is. For the tutorial, you will create a new editable dictionary and then upload terms into it.
4. In the Name field, type Test dictionary and click Save to create the (empty) dictionary.

The new dictionary is created and automatically opened for editing.
5. In the dictionary pane, click Upload.
6. Upload the dictionary-items-organization.csv file from your computer. The terms in the file are uploaded into the dictionary.
7. Click Add Entry to create a new term. An editable row is added at the top of the table.
8. In the Surface Forms column, type IBM and International Business Machines Corporation on separate lines. (When you begin to type a new surface form, a space is added below for an additional surface form.) Leave the radio button next to IBM selected, which indicates that IBM is the lemma.
9. In the Part of Speech column, select Noun.
10. Click Save.

![A screen capture of the Dictionaries page. The "IBM" dictionary entry is selected and its surface forms and part of speech are highlighted.](images/wks_tutdict4.png "A screen capture of the Dictionaries page. The "IBM" dictionary entry is selected and its surface forms and part of speech are highlighted.")







 What to do next 

After you create a dictionary, you can use it to speed up human annotation tasks by pre-annotating the documents.







 Next steps 

Learn how to [create a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_intro).





 Tutorial summary 

While learning about Knowledge Studio, you created a workspace and added artifacts to it.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:
From the [IBM Cloud resource list](https://cloud.ibm.com/resources), select the Storage to view the provisioned IBM Cloud Object Storage instance.
2. For more information, to create IBM Cloud Object Storage bucket, see [create some buckets to store your data](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storagegs-create-buckets).



You used the built-in Terraform capabilities of Schematics to create an IBM Cloud Object Storage service instance in your IBM Cloud account.







 What's next? 

Now that you created your first IBM Cloud resource with Schematics, you can explore the following options:



* Try out this [IBM-provided template](https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-cos-bucket) to create a bucket in the IBM Cloud Object Storage instance that you created with Schematics.
* Learn how to [create your own Terraform template](https://cloud.ibm.com/docs/schematics?topic=schematics-create-tf-config).
* Explore other [IBM-provided templates](https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples).
* Set up the Schematics [CLI](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cli) or [API](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-api) to start automating IBM Cloud resources.
{"id":"my-schema","type":"AVRO","version":1,"createdBy":"","createdOn":1579267788258,"modifiedBy":"","modifiedOn":1579267788258,"globalId":75}

Creating a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Writer role access to the schema resource that matches the schema that is created.



An activity tracker event is generated to report the action. For more information, see [Activity Tracker events](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-at_eventsevents).





 List schemas 

You can generate a list of the IDs of all of the schemas that are stored in the registry by making a GET request to the /artifacts endpoint.

Example curl request:

curl -u token:$APIKEY $URL/artifacts

Example response:

["my-schema"]

Listing schemas requires at least:



* Reader role access to the Event Streams cluster resource type.







 Delete a schema 

Schemas are deleted from the registry by issuing a DELETE request to the /artifacts/{schema-id} endpoint (where {schema-id} is the ID of the schema). If successful, an empty response and a status code of 204 (no content) is returned.

Example curl request:

curl -u token:$APIKEY -X DELETE $URL/artifacts/my-schema

Deleting a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Manager role access to the schema resource that matches the schema that is deleted.



An activity tracker event is generated to report the action. For more information, see [Activity Tracker events](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-at_eventsevents).





 Create a new version of a schema
See [Managing IBM Cloud resources with Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) to create, update, or delete IBM Cloud resources with Terraform.







 Creating a workspace using a Terraform template 



1. Follow the steps in [Setting up Terraform for Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-terraform-setup) to create your workspace with Terraform.
2. See [Managing IBM Cloud resources with Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) to create, update, or delete IBM Cloud resources with Terraform.







 Next steps 

The next stage of working with workspaces is [deploying workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-deploy-wks).
* From the workspace Resources page, find the virtual server and PostgreSQL instance that were created for you.
* Click the link to see the details of your instances.



7. Optional: Remove your Schematics Workspaces and all related IBM Cloud resources.



* Select the Actions drop down list, click Delete.
* Select the Delete workspace and Delete all associated resources option.
* Enter the name of your workspace, and click Delete.





You used the capabilities of Schematics to provision IBM Cloud infrastructure and database services, and automatically configured your services to allow network communication.





 What's next? 



* Explore the capabilities of [IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started).
* Browse other [software and infrastructure templates](https://cloud.ibm.com/catalog?search=label%3Aterraformsoftware) that you can apply with Schematics.
* Learn more about the [built-in capabilities in Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-learn-about-schematics).
* Import your Schematics templates and create your [private IBM Cloud catalog](https://cloud.ibm.com/docs/account?topic=account-create-private-catalog).
* Set up the Schematics [CLI](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cli) or [API](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-api) to start automating the provisioning and management of IBM Cloud resources.
To create a temporal table, set DATA_VERSION_RETENTION_TIME to a nonzero value.

For detailed syntax and privileges, see [the CREATE TABLE command](https://www.ibm.com/docs/en/netezza?topic=npsscr-create-table-2).

CREATE TABLE <tablename> ( <col>[, <col>… ] ) DATA_VERSION_RETENTION_TIME <number-of-days>;

Example:

CREATE TABLE PRODUCT (prodid int, proddesc char(100)) DATA_VERSION_RETENTION_TIME 30;

When you insert a row into the table, the row receives a virtual insert timestamp that is equal to the commit time of the inserting transaction.

When you delete a row from the table, the row receives a virtual delete timestamp that is equal to the commit time of the deleting (or truncating) transaction.





 Creating temporal schemas with the command-line 

To create a temporal schema, set DATA_VERSION_RETENTION_TIME to a nonzero value.

For detailed syntax and privileges, see [the CREATE SCHEMA](https://www.ibm.com/docs/en/netezza?topic=npsscr-create-schema-2) command.

CREATE SCHEMA <schema_name> DATA_VERSION_RETENTION_TIME <number-of-days>;

Example:

CREATE SCHEMA SCHEMA1 DATA_VERSION_RETENTION_TIME 30;





 Creating temporal databases with the command-line 

To create a temporal database, set DATA_VERSION_RETENTION_TIME to a nonzero value.

For detailed syntax and privileges, see [the CREATE DATABASE command](https://www.ibm.com/docs/en/netezza?topic=npsscr-create-database-2).

CREATE DATABASE <db_name> DATA_VERSION_RETENTION_TIME <number-of-days>;

Example:

CREATE DATABASE DB1 DATA_VERSION_RETENTION_TIME 30;







 Creating time travel objects with the web console 



 Creating temporal tables with the web console 



1.
* You have the right permission to create [VPC infrastructure](https://cloud.ibm.com/docs/vpc?topic=vpc-iam-getting-started), [IKS](https://cloud.ibm.com/docs/containers?topic=containers-access_reference) cluster, [LogDNA](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-iam), and [Activity tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-iam) services.



Here are the list of commands used to provision the Agent infrastructure.



* Run [workspace new](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new) command to create the Agent infrastructure workspace.

ibmcloud schematics workspace new --file https://github.com/Cloud-Schematics/schematics-agents/tree/main/tarfiles/create_agent_infra_workspace.json
* Record your workspace ID, and template ID to [upload the tar files](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-upload) provided for installing the Agent infrastructure. Follow the Readme file to extract the .tar file from the [Cloud-Schematics Git repository](https://github.com/Cloud-Schematics/schematics-agents/tree/main/tarfiles).
* Run [workspace upload](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-upload) command to upload the tar file.
[Select schemas](https://cloud.ibm.com/docs-content/v1/content/e2b18e1112dc55facb736744a935365dce13849a/databases-for-enterprisedb/images/emp-select-schemas.png)

Figure 5. Select schemas
4. Select the schemas and click "next".

Zoom

![Select platform](https://cloud.ibm.com/docs-content/v1/content/e2b18e1112dc55facb736744a935365dce13849a/databases-for-enterprisedb/images/emp-select-platform.png)

Figure 6. Select platform
5. Click next to use an existing cluster (or click Go to IBM Cloud to create a new cluster).

Zoom

![Create or use existing cluster](https://cloud.ibm.com/docs-content/v1/content/e2b18e1112dc55facb736744a935365dce13849a/databases-for-enterprisedb/images/emp-cluster.png)

Figure 6. Create or use existing cluster
6. Connect to an existing cluster. Enter the target deployment's details (hostname and port are available on the deployment's endpoints PostgresSQL tab) and click Test Connection to validate the connection. To deploy, click "next" after a successful connection is confirmed.

Zoom

![Connection details for an existing cluster](https://cloud.ibm.com/docs-content/v1/content/e2b18e1112dc55facb736744a935365dce13849a/databases-for-enterprisedb/images/emp-connect-existing.png)

Figure 7. Connection details for an existing cluster

You can use the admin user (after changing its password), or you can create a new db user on the database and use those credentials.
7.
Migration logs have been saved to /root/.enterprisedb/migration-toolkit/logs

 Migration Summary 
Tables: 12 out of 12

Total objects: 12
Successful count: 12
Failed count: 0
Invalid count: 0


Show more









 Oracle to Databases for EnterpriseDB Migration by using MTK only 

You can run schema extraction, schema migration, and data migration by using Migration Toolkit (MTK) only. To do that you need to:



1. Install and setup MTK locally following the steps noted under the heading: [Install EnterpriseDB Migration toolkit](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-oracle-migratinginstall-enterprisedb-migrationtoolkit)
2. Follow the MTK command options for Import Options and Schema Creation to extract and migrate Oracle schema and data [here](https://www.enterprisedb.com/edb-docs/d/edb-postgres-migration-toolkit/user-guides/user-guide/53.0.2/mtk_command_options.html)
3. MTK also supports offline migration for both schema and data [here](https://www.enterprisedb.com/edb-docs/d/edb-postgres-migration-toolkit/user-guides/user-guide/53.0.2/mtk_command_options.htmloffline-migration-options)