vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Getting started with the catalog 

Each instance of IBM Cloud® Data Engine includes a database catalog that you can use to register and manage table definitions for your data on IBM Cloud® Object Storage. Catalog syntax is compatible with Hive metastore syntax. See how to [work with the catalog](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting_started_catalogusage) and refer to the [Catalog management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencechapterHiveCatalog) section of the SQL reference.



 Benefits 

You can explore, change, or discover structured data on [Cloud Object Storage](https://cloud.ibm.com/docs/services/cloud-object-storage/getting-started.htmlgetting-started-console) with Data Engine by using SQL syntax. To query data on Object Storage without a table in the catalog, you need to specify the data location (the corresponding Object Storage URI) and the data format in your SELECT statement. During query execution, data and schema are dynamically discovered as part of the SQL compilation process. This process, called inference, derives column names, data types, the list of partitions, and individual objects on Object Storage that together make up the table data.

Inferring all this information and doing it repetitively with every query imposes latency to your queries. The inference process can take up a significant amount of time, especially for text formats (for example, CSV and JSON), or when thousands of objects exist in different table partitions. In some cases, the inference process even accounts for the largest part of the overall query execution time. So, if you are either familiar with the schema, or want to repetitively use the data for queries, create a table in the catalog. Such a table improves performance for repeated query executions.

Another advantage of creating a table in the catalog is that the table name serves as an alias and is decoupled from the data location. Hence, you can separate the tasks of data engineers and SQL authors. Data engineers deal with the data location and publish registered tables in the catalog by using descriptive table names.
For supported data types, see the [ibmcloud schematics workspace new command](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new). 



2. Create the workspace.

ibmcloud schematics workspace new --file workspace.json
3. Verify that your workspace is created. Make sure that your workspace is in an Inactive state.

ibmcloud schematics workspace list
4. Refer to, [Managing IBM Cloud resources with Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) to start creating, updating, or deleting IBM Cloud resources with Terraform.







 Creating a workspace using the API 



1. Follow the [steps](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-apics_api) to retrieve your IAM access token and authenticate with IBM Cloud Schematics by using the API.
2. Create the workspace.

curl --request POST --url https://schematics.cloud.ibm.com/v1/workspaces -H "Authorization: <iam_access_token>" -d '{"name": "<workspace_name>","type": ["<terraform_version>"],"location": "<location>","description": "<description>","template_repo": {"url": "<github_source_repo_url>"},"template_data": [{"folder": ".","type": "<terraform_version>","variablestore": {"value": "<variable_value>","name": "<variable_name>","type": "<variable_type>","secure": true}]}]}'



JSON file component description

 Parameter Description
* Install or update the [Schematics plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cliinstall-schematics-plugin) version that is greater than the 1.12.5.
* Select the IBM Cloud region that you wish to use to manage your Schematics. Set the region by running [ibmcloud target -r <region>](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_cliibmcloud_target) command. For more information, see [FAQ](https://cloud.ibm.com/docs/schematics?topic=schematics-blueprints-faqfaqs-bp-target-region).
* Check that you have the [IAM permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-accessblueprint-permissions) to create blueprints.



Syntax to create blueprint config using CLI parameters:
They cannot be used interchangeably because they list the author and title fields in a different order. Data encoded with Schema 1 would not be decoded correctly if the decoding process used Schema 2.

When you use the SerDes to create the new schema in the order Schema 1, Schema 2, and Schema 3, the result would be two new schemas. Schema 1 and Schema 2 are different but Schema 3 is the equivalent of Schema 2.

When you create schemas by using the REST API, schemas are considered matching only if they are textually the same, including all attribute ordering and descriptive fields. This is to allow for the case where you want Schema 3 to be a different schema.







 Enabling the Schema Registry 

The Schema Registry is enabled by default for Event Streams Enterprise plan service instances.

The Schema Registry is not automatically enabled for Event Streams Satellite plan service instances. For information about how to enable it for the Satellite plan, see [Enable the Schema Registry API](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-satellite-provisioningsatellite-enable-schema-registry).

The Schema Registry is not available for other Event Streams plans.





 Accessing the Schema Registry 

To access the Schema Registry, you need the URL of the Schema Registry and a set of credentials that can be used to authenticate with the registry. You can find both by inspecting the service credentials of your service. To view these credentials in the UI, click your service instance, select Service Credentials in the left navigation pane, then click the View Credentials link located next to one of the service credentials listed in the table:

Zoom

![Service credentials diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/schema_registry8.png)

Kafka credentials block
('BigCorp', 'Apex'),
('Initech', 'Dallas'),
('Acme Fake Company Names', 'Somewhere');

-- Use the table to augment the Company view with location
-- information.
create view CompanyLoc as
select N2C.location as loc, C.company as company
from Company C, NameToLocation N2C
where Equals(GetText(C.company), GetText(N2C.name));

output view CompanyLoc;
Show more

Splitting a document into smaller subdocuments.

For example, if the main document is a blog that consists of multiple blog entries, you can use GetText to create a subdocument for each blog entry.

detag Document.text as DetaggedBlog
annotate
element 'blog' as Blog
with attribute 'name' as title;

create view BlogEntry as
select B.match as entry, B.title as title
from Blog B;

-- Turn each tuple in the BlogEntry view into a sub-document
create view BlogEntryDoc as
select GetText(B.title) as title, GetText(B.entry) as body
from BlogEntry B;

output view BlogEntryDoc;

--Dictionary for Companies
create dictionary CompanyNameDict as
(
'A Corporation', 'B Corporation'
);

-- Run an extraction over the sub-documents.
-- The spans that this "extract" statement creates will have
-- offsets relative to the blog entries themselves, as opposed
-- to the original multi-entry document.
create view CompanyName as
extract dictionary 'CompanyNameDict' on B.body as name
from BlogEntryDoc B;

output view CompanyName;
Show more





 LeftContext and RightContext 

The LeftContext function takes a Span and an Integer as input:

LeftContext(<input span>, <nchars>)
"description": ""
}
],
"has_githubtoken": false
}
],
"runtime_data": [
{
"id": "3e68db7f-0bea-4d",
"engine_name": "terraform",
"engine_version": "v0.14.11",
"state_store_url": "https://us.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.341cedd5/runtime_data/3e68db7f-0bea-4d/state_store",
"log_store_url": "https://us.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.341cedd5/runtime_data/3e68db7f-0bea-4d/log_store"
}
],
"shared_data": {
"resource_group_id": ""
},
"applied_shareddata_ids": null,
"updated_at": "0001-01-01T00:00:00Z",
"last_health_check_at": "0001-01-01T00:00:00Z",
"cart_id": ""
}
 INFO:root:Completed Creating Schematic Workspace







 Next steps 

After you've successfully created a workspace, you can begin [Generating a plan](https://cloud.ibm.com/docs/hpc-openshift?topic=hpc-openshift-generate-plan&interface=api) to validate all of the configuration properties.
Adding a database-catalog pair 

A database is one of the data sources that you can register and use in IBM® watsonx.data. A catalog defines the schemas and metadata for a data source.

When you add your own object storage bucket or database, or query the data in these data sources through the query engines of watsonx.data, egress charges for pulling data out of these sources might apply depending on your service provider. If you are using managed services, consult your service provider's documentation or support for details about these charges.

To reduce the latency issues, it is recommended to colocate your additional object storage buckets or databases in the region where watsonx.data instance is provisioned.

To add a database-catalog pair, complete the following steps.



1.  Log in to the watsonx.data console.
2.  From the navigation menu, select Infrastructure manager.
3.  To add a database, click Add component and select Add database.
4.  In the Add database window, enter the following details:

Two databases with the same name cannot be added.



Table 1. Register database

 Field                          Description                                                                                   

 Database type                  Select the database type from the list.                                                       
 Database name                  Enter the name of your database.                                                              
 Hostname                       Enter the hostname.                                                                           
 Port                           Enter the port number.                                                                        
 Username                       Enter the port username.                                                                      
 Password                       Enter the password.                                                                           
 Associated catalog definition  Enter the name of the catalog. This catalog is automatically associated with your database.   



5.  Click Add.
The indexes store summary metadata for each partition of your table to avoid scanning data that is not needed for the query execution. For more information, see [index management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-index_management).



 Create index 



 createIndex 

Create an index on the objects in the specified Object Storage location or on the specified table. Define the required index type for each column that you want to calculate the summary metadata for. Create the index on columns that are used for predicates in the SQL statements.





 metaindexIndextype 



* MINMAX: Stores minimum or maximum values for a column for all types, except for complex types.
* VALUELIST: Stores the list of unique values for the column for all types if the distinct values in that column are low.
* BLOOMFILTER: Uses bloom filter technique for byte, string, long, integer, or short types if the distinct values in that column are high.
* GEOSPATIAL: Stores a geospatial bounding box for geometry types.



-- create an index on the columns temp, lat, lng, vid and city of the metergen sample table
CREATE METAINDEX
MINMAX FOR temp,
MINMAX FOR lat,
MINMAX FOR lng,
BLOOMFILTER FOR vid,
VALUELIST FOR city
ON cos://us-geo/sql/metergen STORED AS parquet

-- create an index on the columns customerID and city of the sample table CUSTOMERS_PARTITIONED
CREATE METAINDEX
VALUELIST for city,
BLOOMFILTER for customerID
ON TABLE CUSTOMERS_PARTITIONED

-- create a geospatial index on the column location of the hospitals sample table
CREATE METAINDEX
GEOSPATIAL FOR location
ON cos://us-geo/sql/hospitals_geometry.parquet STORED AS parquet

Before you start to use data skipping index management commands, ensure that you set the base location in Object Storage, where you want the metadata to be stored. Use the following command:

-- set the default location for all indexes
{"id":"my-schema","type":"AVRO","version":2,"createdBy":"","createdOn": 1579267978382,"modifiedBy":"","modifiedOn":1579267978382,"globalId":83}

Creating a new version of a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Writer role access to the schema resource that matches the schema that gets a new version.



An activity tracker event is generated to report the action. For more information, see [Activity Tracker events](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-at_eventsevents).





 Get the latest version of a schema 

To retrieve the latest version of a particular schema, make a GET request to the /artifacts/{schema-id} endpoint (where {schema-id} is the ID of the schema). If successful, the latest version of the schema is returned in the payload of the response.

Example curl request:

curl -u token:$APIKEY $URL/artifacts/my-schema

Example response:

{"type":"record","name":"Citizen","fields":[{"name": "firstName","type":"string"},{"name":"lastName","type":"string"},{"name":"age","type":"int"},{"name":"phoneNumber","type":"string"}]}

Getting the latest version of a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Reader role access to the schema resource that matches the schema that is retrieved.







 Getting a specific version of a schema
The <table name> part specifies the table that is created in your database. It has the format <schemaname>.<tablename>. If you omit the <schemaname> part, the table is created in the schema of database user that was created for the IBMid of the SQL user. The table name is case-preserving, so use uppercase to match database defaults.

The following URI is an example of a Db2 table URI:

db2://db2w-vqplkwx.us-south.db2w.cloud.ibm.com/MYSCHEMA.QUERY_RESULT
In this section, you will create the following cloud services required for the application using IBM Cloud Schematics: IBM Cloud Databases for PostgreSQL and IBM Cloud Object Storage.



1. Navigate to [Schematics Workspaces](https://cloud.ibm.com/schematics/workspaces), click on Create workspace.



1. Under the Specify Template section, provide https://github.com/IBM-Cloud/vpc-scaling-dedicated-host under GitHub or GitLab repository URL.
2. Select terraform_v1.2 as the Terraform version and click Next.



2. Under Workspace details,



1. Provide a workspace name : vpc-scaling-workspace.
2. Choose a Resource Group and a Location.
3. Click on Next.



3. Verify the details and then click on Create.
4. Under Variables, set step1_create_services to true by clicking the action menu in the row > Edit, uncheck Use default, choose true from the Override Value dropdown, and click on Save.
5. Set any additional variables you would like to override, the most typical ones are region, resource_group_name.
6. Scroll to the top of the page and click Generate plan. This is the same as terraform plan command.
7. Click on Show more to check the resources to be provisioned.
8. Navigate to the workspace page using the breadcrumb menu and click on Apply plan. Check the logs to see the status of the services created.



Navigate to the [resource list](https://cloud.ibm.com/resources). Here, you can filter by the basename used to create the resources, i.e., vpc-scaling, and you will see the cloud services required for this tutorial provisioned in the resource group you specified. All the data stored with these services is encrypted with a key generated and stored in IBM Key Protect for IBM Cloud.



 Enable logging and monitoring 

You can have multiple IBM Cloud Log Analysis instances in a location.
"value": "",
"type": "",
"description": ""
}
],
"has_githubtoken": false
}
],
"runtime_data": [
{
"id": "3e68db7f-0bea-4d",
"engine_name": "terraform",
"engine_version": "v0.14.11",
"state_store_url": "https://us.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.341cedd5/runtime_data/3e68db7f-0bea-4d/state_store",
"log_store_url": "https://us.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.341cedd5/runtime_data/3e68db7f-0bea-4d/log_store"
}
],
"shared_data": {
"resource_group_id": ""
},
"applied_shareddata_ids": null,
"updated_at": "0001-01-01T00:00:00Z",
"last_health_check_at": "0001-01-01T00:00:00Z",
"cart_id": ""
}
 INFO:root:Completed Creating Schematic Workspace







 Next steps 

After you've successfully created a workspace, you can begin [Generating a plan](https://cloud.ibm.com/docs/ibm-spectrum-lsf?topic=ibm-spectrum-lsf-generate-plan&interface=api) to validate all of the configuration properties.
For example, to install pg_stat_statements on the ibmclouddb database,

ibmclouddb=> CREATE EXTENSION pg_stat_statements;
CREATE EXTENSION

Extensions are installed into the read-only ibm_extension schema. The schema is part of the search_path so extension objects do not need to be qualified with a schema. The change from public schema to ibm_extension schema is necessary to protect the security and integrity of your data.

If you run the dx command after installing an extension, it appears in the table.

ibmclouddb=> dx
List of installed extensions
Name | Version | Schema | Description
--------------------+---------+---------------+-----------------------------------------------------------
pg_stat_statements | 1.5 | ibm_extension | track execution statistics of all SQL statements executed
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(2 rows)

Database extensions in PostgreSQL are managed per database. If you have multiple databases that you need to install an extension on, run the CREATE command on each database.





 Upgrading Extensions 

If there is a newer version of an extension available than the one you currently have installed, use the ALTER EXTENSION to upgrade it.





 Extension-Specific Notes 



 pg_repack 



* [The pg_repack documentation](http://reorg.github.io/pg_repack/)
* When you run the pg_repack command, pass the -k flag in to bypass the check for superuser. For example,

pg_repack -k [OPTION]... [DBNAME]
* For pg_repack to run reliably, your deployment should be on PostgreSQL 9.6 and above.
* If you do not have a project or do not want to use any of the projects that are listed, click Create a new project to add one. Follow the procedure in [Create a data collection](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addsearch-skill-add-create-discovery-collection).









 Create a data collection 

When the Discovery application opens in a new browser tab or window, the Projects page might be displayed briefly. You do not need to create a project. A project is created for your collection automatically. Wait for the collections page to be displayed.



1. To create a Discovery collection, do one of the following things:



* To create a collection from data that is stored in a type of data source for which Discovery provides built-in support, pick a data source type.

![Shows the types of data collections you can create](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/discovery-data-collections.png)



1. Click Next.
2. Provide the required information for the data source you choose.

See [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections) for more details.
3. Indicate the frequency with which you want data from the data source to be synchronized with the collection you are creating in Discovery.
4. Specify the information that you want to extract from the data source and include in your Discovery collection.

The options that are displayed differ depending on the data source type.



* For a Salesforce data source, you select the object types that you want to extract from the source documents. You might select a [Case object type](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_case.htm!) that represents a case, which is a customer issue or problem, for example.
Zoom

![Shows the options for creating a facet.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-facet-from-field.png)

Figure 34. Creating a facet
9. Choose the index field that is associated with the dictionary enrichment that you applied to the subtitle field. From the Field field, select enriched_subsection.entities.mentions.text

Zoom

![Shows the options for the facet field.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-facet-field.png)

Figure 35. Fields from which you can create a facet

You might need to hover over the entries to see the full field names.
10. Add a label, such as Dictionary terms to the Label field, and then click Apply.

Zoom

![Shows the completed facet creation panel.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-facet-complete.png)

Figure 36. Facet was created
11. Enter Where do muni bond trades get reported to? as the search query.

The Dictionary terms facet that you created is displayed along with the search results. A Municipal Bonds checkbox is shown, which indicates that at least one of the returned passages is extracted from a document segment with the term Municipal Bonds in its subtitle field.

Zoom

![Shows the Dictionary terms facet with a Municipal Bonds option.]
[List of properties](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-properties-default.png)

Figure 1. List of properties



 Create a property 

To create a property, complete these steps:



1. In the App Configuration console, click Properties.
2. Select the Current Environment to display the list of properties in that environment.
3. Click Create. The side panel opens with fields for creating a new property.

Zoom

![Create property](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-create-property.png)

Figure 2. Creating a new property
4. Provide the property details:



* Name - name of the property.
* Property ID - the property ID value is auto suggested based on the entered property name. You can modify the same, if needed. Use the Property ID value as the identifier in your SDK or API code.
* Property type - specify the type of the property. Supported type includes Boolean, String, Numeric, and Secret reference. For more information about property type and default values, see [Selecting property type](https://cloud.ibm.com/docs/app-configuration?topic=app-configuration-ac-propertiesselecting-properties-type).
* Default value - specify the default value for the property type selected. For more information, see [Selecting property type](https://cloud.ibm.com/docs/app-configuration?topic=app-configuration-ac-propertiesselecting-properties-type).
* Optionally, you can add the property to one or more collections by selecting in the Add to collection list.
* Description - add a description of the property, which is optional.