vania-janet/multiturn-rag-retrieval-data huggingface dataset:
ibmcloud schematics blueprint create 

Create a blueprint config from the command line using the ibmcloud schematics blueprint create command. Creating a blueprint config is the first step to deploy a blueprint environment. The blueprint config is created from user provided parameters via the command line that specifies the source of the blueprint template in a Git repository, and input values sourced from Git hosted input files and locally sourced input values.

The create command supports passing all required parameters via options on the CLI and passing input values from local files.

A experimental file option is provided to create a blueprint config from a configuration provided as a JSON file from the local file system. Refer to the [ibmcloud schematics blueprint create - file option](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-blueprint-create) section. These two usage modes are mutually exclusive.

For Schematics Blueprints, the [Schematics plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cliinstall-schematics-plugin) version must be greater than the 1.12.5 version.

Refer to the section on [Creating Blueprints](https://cloud.ibm.com/docs/schematics?topic=schematics-create-blueprint-config&interface=cli) for examples of command syntax and output.

Syntax to create blueprint config using CLI parameters:
Identify the data from Netezza Performance Server to merge and compare. 

In this example, data that was loaded into Netezza Performance Server (YELLOW_TAXI_JANUARY_2022) is compared with data from a data like (YELLOW_TAXI_JANUARY_2021).

To follow this example, ensure that YELLOW_TAXI_JANUARY_2022 is in your Netezza Performance Server database.

a) Create an external table for the data that you want to load (YELLOW_TAXI_JANUARY_2022).

Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);

Example:

create EXTERNAL table YELLOW_TAXI_JANUARY_2022 on EXAMPLEDATASOURCE
using ( 
 DATAOBJECT ('/yellow_tripdata_2022-01.parquet') FORMAT 'PARQUET' 
);

b) Load the data (YELLOW_TAXI_JANUARY_2022) into Netezza Performance Server.

create table 'TABLE NAME LOADED' as
   select
        
   from
       'TABLE';

Example:

create TABLE YELLOW_TAXI_JANUARY_2022_LOADED as
   select
        
   from
       YELLOW_TAXI_JANUARY_2022;

INSERT 0 2463931





 3. Create an external table for the data from a data lake. 

Create an external table that accesses the yellow taxi data from January 2021.

Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);

The DATAOBJECT argument must reference a single file in the parquet format.
FULL Latest version of the schema A new version of the schema can add optional fields that are not present in the existing version of the schema. 
 FULL_TRANSITIVE All versions of the schema A new version of the schema can omit optional fields that are present in the existing version of the schema. 



These rules can be applied at two scopes:



1. At a global scope, which is the default that is used when creating a new schema version.
2. At a per-schema level. If a per-schema level rule is defined, it overrides the global default for the particular schema.



By default, the registry has a global compatibility rule setting of NONE. Per-schema level rules must be defined, otherwise the schema defaults to using the global setting.





 Full API description 

For a description of the REST API with examples, see [Event Streams schema-registry-rest](https://github.com/ibm-messaging/event-streams-docs/tree/master/schema-registry-api).

You can download the full specification for the API from the [Event Streams Schema Registry REST API YAML file](https://github.com/ibm-messaging/event-streams-docs/blob/master/schema-registry-api/openapi.yaml). To view the Swagger file, use Swagger tools, for example [Swagger editor](http://editor.swagger.io//).

For more information about accessing the Schema Registry using an SDK, see [Event Streams Schema Registry REST API](https://github.com/IBM/eventstreams-go-sdk/blob/main/schema_operations.md).

For information about Event Streams resources and data sources on Terraform, see [resources and data sources](https://cloud.ibm.com/docs/ibm-cloud-provider-for-terraform?topic=ibm-cloud-provider-for-terraform-resources-datasource-listibm-event-streams_rd).





 Using the Schema Registry with the third-party SerDes
This tutorial walks you through the creation of a web application using the popular MEAN stack. It is composed of a MongoDB, Express web framework, Angular front end framework and a Node.js runtime. You will learn how to run a MEAN sample app locally, create and use a managed database-as-a-service (DBasS), deploy the app to IBM Cloud and scale both the runtime and database resources.

Code Engine Databases For MongoDB



* 1 hour
* 2023-05-05



[Getting Started](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-getting-started)Getting Started

This tutorial is a short introduction to using an IBM Cloud® Databases for MySQL deployment. To get started, create a connection to MySQL Workbench, an open source administration platform for MySQL that provides many tools for managing your data and databases. Download and install the version that is appropriate to your environment and then consult MySQL's documentation to connect and manage your Databases for MySQL deployment.

Databases For MySQL



* 30 minutes
* 2023-04-28



[Getting Started](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started)Getting Started

This tutorial is a short introduction to using an IBM Cloud® Databases for PostgreSQL deployment. pgAdmin is an open source administration platform for PostgreSQL, and provides many tools for managing your data and databases. Download and install the version that is appropriate to your environment, and then follow the steps to connect it to your Databases for PostgreSQL deployment.

Databases For PostgreSQL



* 30 minutes
* 2023-05-01



[Scale workloads in shared and dedicated VPC environments](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-scaling-dedicated-compute)Scale workloads in shared and dedicated VPC environments
* [Planning and applying the Terraform template](https://cloud.ibm.com/docs/schematics?topic=schematics-provisioning-terraform-templatetut-plan-wks)
* [What's next?](https://cloud.ibm.com/docs/schematics?topic=schematics-provisioning-terraform-templatetut_whats_next)







 Setting up continuous deployment with Schematics and DevOps toolchain 

[Setting up continuous deployment with Schematics and DevOps toolchain](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-continuous-deployment)



* [Description](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-desc)
* [Objectives](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-obj)
* [Time needed](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-timereq)
* [Audience](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-tut-audience)
* [Prerequisites](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentworkspace-prereq)
* [Accessing the IBM Cloud and GitHub](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentaccess-automate-template)
* [Creating your IBM Cloud Schematics Workspace](https://cloud.ibm.com/docs/schematics?topic=schematics-workspace-continuous-deploymentcreate-wkspace)
The Schematics ibmcloud terraform command usage displays warning and deprecation message as Alias 'terraform' will be deprecated. Use 'schematics' or 'sch' in your commands.



 Create file template in JSON format 

Schematics supports to download the Terraform modules template from the private repository. For more information, see [Supporting to download modules from private remote host](https://cloud.ibm.com/docs/schematics?topic=schematics-download-modules-pvt-git).

You can create the JSON file as shared in the example.json file for workspace creation and pass the file path along with the file name in --file flag. The description of all the parameters of example.json as described in the table.

You need to replace the <...> placeholders with the actual values. For example, "<workspace_name>" as "testworkspace".

example.json:

{
"name": "<workspace_name>",
"type": [
"<terraform_version>"
],
"location": "<location>",
"resource_group": "<resource_group>",
"description": "<workspace_description>",
"tags": [],
"template_repo": {
"url": "<github_source_repo_url>"
},
"template_data": [
{
"folder": ".",
"type": "<terraform_version>",
"env_values":
{
"VAR1":"<val1>"
},
{
"VAR2":"<val2>"
}
],
"variablestore":
{
"name": "<variable_name_x>",
"value": "<variable_value_x>",
"type": "string",
"secure": true,
"title": "User Profile Standard v2 Schema",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"profile_name": {
"type": "string"
},
"profile_version": {
"type": "string"
},
"controls": {
"type": "array",
"items": {
"type": "object",
"properties": {
"control_specifications": {
"type": "array",
"items": {
"type": "object",
"properties": {
"assessment_count": {
"type": "number"
},
"assessments": {
"type": "array",
"items": {
"type": "object",
"properties": {
"assessment_type": {
"type": "string"
},
"assessment_method": {
"type": "string"
},
"assessment_description": {
"type": "string"
},
"assessment_id": {
"type": "string"
},
"parameter_count": {
"type": "number"
},
"parameters": {
"type": "array",
"items": {
"type": "object",
"properties": {
"parameter_name": {
"type": "string"
},
"parameter_display_name": {
"type": "string"
},
"parameter_type": {
"type": "string"
}
}
}
}
}
}
}
}
}
}
}
}
},
"default_parameters": {
2. Select Start creating.
3. Select a project from the list of available projects. You can also [create a new one](https://cloud.ibm.com/docs/codeengine?topic=codeengine-manage-projectcreate-a-project). Note that you must have a selected project to create a job.
4. Select Job.
5. Enter a name for the job, for example, myjob. Use a name for your job that is unique within the project.
6. Specify icr.io/codeengine/helloworld for the image reference. For this example, you do not need to modify the default values. For more information about the code that is used for this example, see [helloworld](https://github.com/IBM/CodeEngine/tree/main/hello).
7. Click Create.
8. From your job page, click Submit job to submit a job based on the current configuration.
9. From the Submit job pane, accept all the default values, and click Submit job again to run your job.



When logging is enabled, the following example is displayed in the logs. To learn about running jobs with logging enabled, see [Viewing logs](https://cloud.ibm.com/docs/codeengine?topic=codeengine-view-logs).

Example output from logging instance

Hello World from:
. ___ __ ____ ____
./ __)/ ( ( __)
( (__( O )) D ( ) _)
.___)__/(____/(____)
.____ __ _ ___ __ __ _ ____
( __)( ( / __)( )( ( ( __)
.) _) / /( (_ )( / / ) _)
(____)_)__) ___/(__)_)__)(____)
SELECT * FROM cos://us-geo/sql/customers.csv
INTO cos://us-geo/mybucket/customers_partitioned.csv
PARTITIONED BY (country)





 Step 2: Attach table partitions 

After you defined a partitioned table, it is initially empty and you must attach the partitions to it explicitly. A convenient way to add all partitions that exist on Object Storage, is to use the following RECOVER PARTITIONS clause.

ALTER TABLE customers RECOVER PARTITIONS

This clause overwrites the current partition definitions for the table with the structure that is detected from Object Storage data by using the location prefix that is specified for the table. You can also update partition definitions selectively with the ADD PARTITION and DROP PARTITION clauses of the ALTER TABLE statement, for example, to attach more data to a table that was uploaded recently.

When you added all partitions, the partitioned table is set up to be queried. You get all the German customers, if you submit the following query:

SELECT customerID FROM customers WHERE country = 'Germany'

The query execution reads the objects only under the cos://us-geo/sql/customers_partitioned.csv/country=Germany/ prefix because the partition definitions are used by the query optimizer to minimize the necessary data transfer.







 Limitations 



* With the Standard plan, you can create up to 100 tables with up to 20,000 partitions per table.
* If you use the Lite plan, the catalog management features, such as CREATE TABLE, are not allowed.
* The ADD PARTITION option of the ALTER TABLE statement may not correctly locate partitions if the value for a partition column contains special characters, such as the colon : that can appear as a timestamp separator.

When the location is inferred from one or more partition values, some special characters in the values are URL escaped when you construct the Object Storage location. For example, the statement ALTER TABLE mytable ADD PARTITION ( startTime = '2020-01-01 12:00:00' ) constructs a partition with an Object Storage location .../startTime=2020-01-01 12%3A00%3A00/.
* Add tables to publisher.

exampledb=> ALTER PUBLICATION my_publication ADD TABLE my_table;







 Setting up Logical Replication on the Subscriber 

To configure your Databases for EnterpriseDB deployment as a subscriber,



* Log in to the database created for replication with admin user.

psql -U admin -d exampledb
* Run the following query to call the create_subscription function and create the subscriber channel.

exampledb=> SELECT create_subscription('subs1','130.215.223.184','5432','admin','password','exampledb','my_publication');







 Monitoring Replication 

You can monitor the status of logical replication from both the publisher and the subscriber by running the following query on either.

exampledb=> SELECT * FROM pg_stat_replication;
Using Event Streams Schema Registry 

Schema Registry provides a centralized repository for managing and validating schemas. Schemas in a Schema Registry provide the explicit contract that a program generating an event provides to other programs that are consuming those events.



 Schemas overview 

Apache Kafka can handle any data, but it does not validate the information in the messages. However, efficient handling of data often requires that it includes specific information in a certain format. Using schemas, you can define the structure of the data in a message, ensuring that both producers and consumers use the correct structure.

Schemas help producers create data that conforms to a predefined structure, defining the fields that need to be present together with the type of each field. This definition then helps consumers parse that data and interpret it correctly. Event Streams Enterprise plan supports schemas and includes a Schema Registry for using and managing schemas.

It is common for all of the messages on a topic to use the same schema. The key and value of a message can each be described by a schema.

Zoom

![Schemas overview diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/schema_registry1.svg)

Schemas overview





 Schema Registry 

Schemas are stored in the Event Streams Schema Registry. In addition to storing a versioned history of schemas, it provides an interface for retrieving them. Each Enterprise plan Event Streams instance has its own Schema Registry.

Producers and consumers validate the data against the specified schema that is stored in the Schema Registry (in addition to going through Kafka brokers). The schemas do not need to be transferred in the messages this way, meaning the messages can be smaller.

Zoom

![Schema Registry architecture diagram.]
To create a custom dashboard, see [Creating custom dashboard](https://cloud.ibm.com/docs/schematics?topic=schematics-monitoring-instancescreate-dashboard).







 Creating custom dashboard 

You can create your custom dashboard in the IBM Cloud console.

Complete these steps to create your custom dashboard:



1. Log in to your [IBM Cloud](https://cloud.ibm.com) account by using your credentials.
2. From the IBM Cloud page, select Navigation menu > Observability > Monitoring.
3. Click Create Custom Dashboard to view the create dashboard from template.
4. Name your dashboard, and click Create and Open.
5. Click Add Dashboard icon, the New Dashboard page opens.
6. Now, you can use your custom dashboard to edit the metrics that you want to monitor, the counts of your workspace state, action, and its success and failure status.
7. Click Save.

For more information about deleting a dashboard, see [Deleting a dashboard](https://cloud.ibm.com/docs/monitoring?topic=monitoring-removeremove_ui).







 IBM Cloud Schematics metrics details 

Schematics supports three metrics that you can use to configure in your dashboard for monitoring. The tables provide the details about the metrics.



Metrics details

 Metric name Enterprise Lite Standard 

 [ibm_schematics_workspace_actions_count](https://cloud.ibm.com/docs/schematics?topic=schematics-monitoring-instanceswkspace-actions-count) Yes No Yes 
 [ibm_schematics_workspace_count](https://cloud.ibm.com/docs/schematics?topic=schematics-monitoring-instanceswkspace-actions-count) Yes No Yes 
 [ibm_schematics_workspace_vulnerability_count](https://cloud.ibm.com/docs/schematics?topic=schematics-monitoring-instanceswkspace-vulnerability-count) Yes No Yes 





 ibm_schematics_workspace_count
"cart_id": "",
"last_action_name": "WORKSPACE_CREATE",
"last_activity_id": "e27b2c73cad6571d4abee1092065076d",
"last_job": {
"job_id": "e27b2c73cad6571d4abee1092065076d",
"job_name": "WORKSPACE_CREATE",
"job_status": ""
}
},
]
}

On successful workspace list, it returns the list of the workspace.

For more information, see [troubleshooting section](https://cloud.ibm.com/docs/schematics?topic=schematics-wks-create-api&interface=cli).







 Displaying the workspace list with Terraform 



1. Follow the steps in [Setting up Terraform for Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-terraform-setup) to create your workspace with Terraform.
2. See [Managing IBM Cloud resources with Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) to create, update, or delete IBM Cloud resources with Terraform.







 Next steps 

The next stage of working with workspace is [deploying workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-deploy-wks).
The create external view statement 

The create external view statement in AQL allows specification of more metadata about a document as a new view, in addition to the predefined Document view that holds the textual and label content.



 Syntax 

create external view <view_name> (
<colname> <type> [, <colname> <type>]
)
external_name '<view_external_name>';





 Description 



* <view_name>

Specifies the internal name of the external view. The external view is referred by this name in the AQL rules. A <view_name> can be a simple identifier or a double-quoted identifier. A <view_name> cannot contain the period character.
* <colname>

Specifies the name of the column to define in the external view.
* <type>

Specifies the data type for the associated column. The data types that are supported for external view columns are Text, Span, Integer, and Float.
* '<view_external_name>'

Specifies the external name of the external view. External systems that populate tuples into the external view refer to the external view by the external name. The '<view_external_name>' must be a String constant that is encased in single quotation marks ('ExternalName').







 Examples 

To illustrate external views, consider an example application that requires that you identify the names of persons in email messages.

Example 1: Identifying the names of persons in email messages

Assume that the text of an email message is "Ena, please send me the document ASAP". While a human might be able to understand that Ena is a name of a person based on the text of the email, AQL rules that are written to identify names of persons with high precision from general text might be too conservative and not able to draw the same conclusion with high confidence, based on the fact that Ena is a capitalized word.

One way to boost the coverage of the rules is to use words from the From, To, and CC fields of the email as more evidence.
Creating custom entities and relations models 

The Natural Language Understanding Free plan limits the size and performance of your custom model. To test a custom model to its full extent, use it with the Natural Language Understanding Standard plan.



1. [Get started with Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutintrowks_tutintro).
2. Create a custom model.



1. To create a custom entities and relations model, see [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_intro)
2. You can also create a custom entities model with a rule-based model. See [Creating a rule-based model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutrule_intro) for details.



3. [Deploy your model to Natural Language Understanding](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_manlu)
4. To use your model, specify the model that you deployed in the [entities](https://cloud.ibm.com/apidocs/natural-language-understandingentities) or [relations](https://cloud.ibm.com/apidocs/natural-language-understandingrelations) options of your API request:



* Example parameters.json file:

{
"url": "www.url.example",
"features": {
"entities": {
"model": "your-model-id-here"
},
"relations": {
"model": "your-model-id-here"
}
}
}
* Example curl request: