vania-janet/multiturn-rag-retrieval-data huggingface dataset:
A summary schema MUST be a [graph fragment](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-schemasgraph-fragment-pattern) variant of the canonical schema for the resource type.





 Resource creation and replacement schemas 

The request schema used in an operation to create a new resource with a POST (or PUT) request or to replace a resource

[1]with a PUT request SHOULD be the name of the canonical schema for the resource type, suffixed with Prototype.

For example, a POST /bulldozers operation would accept the information needed to create a new bulldozer represented with the BulldozerPrototype schema.

A prototype schema MUST be a [graph fragment](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-schemasgraph-fragment-pattern) variant of the canonical schema except where a property is required by a domain-specific constraint to be explicitly write-only.

There is an important exception to this guidance if the nature of an API is to provide clients with full control over entire resources, with no system-defined or immutable aspects. For such APIs, the canonical schema itself SHOULD be used in requests to create or replace a resource.





 Resource reference schemas 

When a related resource is referenced within another resource's schema, properties of the related resource SHOULD be represented with a schema with the name of the related resource's canonical schema, suffixed with Reference. The property that a resource reference is embedded in MAY be named for the related resource type or (especially if the relationship is polymorphic) the nature of the relationship.

A resource reference MUST be a [graph fragment](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-schemasgraph-fragment-pattern) variant of the canonical schema (and, if it exists, the summary schema) for the resource type except for properties that indicate a resource no longer exists.

The resource's unique, immutable handle (usually its id) as represented in the final path parameter of its canonical URI MUST be included in its reference.
The tool displays a number of messages as it creates the project directory and adds a number of directories and files to it.It also runs npm install to install all the project dependencies, as specified in package.json. This process might take some time.
5. Before you can create a model you need to make sure that your current working directory is the project top-level directory. To do this, enter the following command:



cd <project directory name>

Note: Making your current working directory the project top-level directory will ensure that the option to add models and data sources is available when you open the API Designer. If you open the API Designer without changing the directory, you will not be able to add models and data sources.

To open the API Designer, complete the following steps:



1. Open a command line and enter:



apic edit

After a brief pause, the following message is displayed.

Express server listening on http://127.0.0.1:9000

The API Designer opens in your default web browser.



2. In the login page, enter your IBM Cloud ID and password. Click Sign In.
3. Click Models.
4. Click Add. The LoopBack Model window opens.
5. Enter a model name. In general, you can use any alphanumeric characters plus dashes and underscores in a model name.
6. Click New.
7. In the model editor page, go to Properties and click Add.
8. Enter the Property Name and select a Type
9. When you finish entering the Properties for the model, click Save.



By default, an empty LoopBack project does not have any data sources defined. To define a data source, complete the following steps:



1. Click Data Sources.
2. Click the Add icon. The New LoopBack Model window opens.
3. Enter a data source name. You can use any alphanumeric character, dashes, and underscores in a data source name.
4. Click New.
"error_code":404,
"message":"No artifact with id 'my-schema' might be found."
}

The properties of the error JSON object are as follows:



 Property name Description 

 Error_code The HTTP status code of the response. 
 Message A description of the cause of the problem. 
 Incident This field is only included if the error is a result of a problem with the Schema Registry. This value can be used by IBM service to correlate a request to diagnostic information captured by the registry. 







 Create a schema 

This endpoint is used to store a schema in the registry. The schema data is sent as the body of the POST request. An ID for the schema can be included by using the ‘X-Registry-ArtifactId' request header. If this header is not present in the request, an ID is generated. The content type header must be set to “application/json”.

Example curl request:

curl -u token:$APIKEY -H 'Content-Type: application/json' -H 'X-Registry-ArtifactId: my-schema' $URL/artifacts -d '{"type":"record","name":"Citizen","fields":[{"name": "firstName","type":"string"},{"name":"lastName","type":"string"},{"name":"age","type":"int"},{"name":"phoneNumber","type":"string"}]}'

Example response:

{"id":"my-schema","type":"AVRO","version":1,"createdBy":"","createdOn":1579267788258,"modifiedBy":"","modifiedOn":1579267788258,"globalId":75}

Creating a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Writer role access to the schema resource that matches the schema that is created.
Schemas 

Object schemas, the expression of models in OpenAPI, generally represent a resource.



 Naming 

Schema names SHOULD be nouns or compound nouns represented in upper camel case. Schema names MUST be simple, descriptive, and meaningful to developers.



 Canonical resource representation 

The schema for the canonical representation of a resource in a response SHOULD correspond directly to the path segment(s) representing the resource type and to [the way the resource type is referenced](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-operations) in the operation ID.

For example, a GET /boats/{id} operation would return a resource represented with the Boat schema and a GET /boats/{boat_id}/oars/{id} operation would return a resource represented with the BoatOar schema.





 The graph fragment pattern 

Non-canonical variations of a resource schema, such as those used to create or refer to a resource, SHOULD be "graph fragment" variants of the schema. A graph fragment schema has the same structure as its canonical schema with some properties omitted from the schema or from any nested object schemas. For example, if a schema defines objects such as the following:

{
"name": "Stanley Q. Dallas",
"birthdate": "1955-10-19",
"address": {
"street": "55 West Oak Grove",
"city": "Happenstance",
"state": "CT",
"postal_code": "06058"
}
}

A graph fragment variant of the same schema might define objects such as:

{
"name": "Stanley Q. Dallas",
"address": {
"postal_code": "06058"
}
}





 Resource collection schemas
You will find instructions to download and install these tools for your operating environment in the [Getting started with tutorials](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-tutorials) guide.



1. Clone the [Github repository for this tutorial](https://github.com/IBM-Cloud/cloud-sql-database) and change into its directory. In a terminal, execute the following lines:

git clone https://github.com/IBM-Cloud/cloud-sql-database.git
cd cloud-sql-database
2. Go to [GeoNames](https://www.geonames.org/) and download and extract the file [cities1000.zip](https://download.geonames.org/export/dump/cities1000.zip). It holds information about cities with a population of more than 1000. You are going to use it as data set.







 Step 1: Provision the SQL Database 

Start by creating an instance of the [Db2 Warehouse on Cloud](https://cloud.ibm.com/catalog/services/db2-warehouse) service.



1. Visit the [IBM Cloud® console](https://cloud.ibm.com/). Click on Catalog in the top navigation bar.
2. Click on Databases on the left pane and select Db2 Warehouse.
3. Pick the Flex One plan and change the suggested service name to sqldatabase (you will use that name later on). Pick a resource group and a location for the deployment of the database.
4. Click on Create. The provisioning starts.
5. In the Resource List, locate the new instance under Databases and wait for it to be available (sometimes you may need to refresh the page). Click on the entry for your Db2 Warehouse on Cloud service.
6. Click on Open Console to launch the database console.







 Step 2: Create a table 

You need a table to hold the sample data. Create it using the console.



1.
Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);

The DATAOBJECT argument must reference a single file in the parquet format. If you want to query from multiple parquet files, you must create additional external tables.

Example:

create EXTERNAL table YELLOW_TAXI_JANUARY_2021 on EXAMPLEDATABASE 
using ( 
 DATAOBJECT ('/yellow_tripdata_2021-01.parquet') FORMAT 'PARQUET' 
);





 4. Query the remote and loaded data. 

Now, you can query both the local 2022 data that was loaded and the 2021 data from a data lake all in the same query.

The parquet column names are case sensitive. You must use double quotation marks ("") when you are querying specific columns.



* To identify which year had the most passengers, run:

select
(
select
sum("passenger_count")
from
YELLOW_TAXI_JANUARY_2022_LOADED) as "passengers 2022",
(
select
sum("passenger_count")
from
YELLOW_TAXI_JANUARY_2021) as "passengers 2021";

Output:

passengers 2022 | passengers 2021
-----------------+-----------------
3324167 |     1794615
(1 row)
* To compare how many passengers travelled between 1:00 AM and 6:00 PM in 2021 and 2022, run:

select
(
select
sum("passenger_count")
from
YELLOW_TAXI_JANUARY_2022_LOADED
where
"tpep_pickup_datetime"::time > '1:00am'
Now run the following script called postgres-parquet-table-create.py to create a Parquet table with metadata from IBM Cloud Object Storage in the Databases for PostgreSQL database.

from pyspark.sql import SparkSession
import time
def init_spark():
  spark = SparkSession.builder.appName("postgres-create-parquet-table-test").getOrCreate()
  sc = spark.sparkContext
  return spark,sc
def generate_and_store_data(spark,sc):
  data =[("1","Romania","Bucharest","81"),("2","France","Paris","78"),("3","Lithuania","Vilnius","60"),("4","Sweden","Stockholm","58"),("5","Switzerland","Bern","51")]
  columns=["Ranking","Country","Capital","BroadBandSpeed"]
  df=spark.createDataFrame(data,columns)
  df.write.parquet("cos://<CHANGEME-BUCKET>.mycosservice/broadbandspeed")
def create_table_from_data(spark,sc):
  spark.sql("CREATE TABLE MYPARQUETBBSPEED (Ranking STRING, Country STRING, Capital STRING, BroadBandSpeed STRING) STORED AS PARQUET  location 'cos://CHANGEME-BUCKET.mycosservice/broadbandspeed/'")
  df2=spark.sql("SELECT * from MYPARQUETBBSPEED")
  df2.show()
def main():
  spark,sc = init_spark()
  generate_and_store_data(spark,sc)
  create_table_from_data(spark,sc)
  time.sleep(30)
if __name__ == '__main__':
  main()
Show more
7.
Software deployment in IBM Cloud Schematics 

Try out one of the IBM® provided software templates to quickly spin up a classic Virtual Server Instance (VSI), and automatically configure the instance to connect to an IBM Cloud® Databases for PostgreSQL instance.

With IBM Cloud Schematics, you can choose from a wide variety of [software and infrastructure templates](https://cloud.ibm.com/catalogsoftware) that you can use to set up IBM Cloud services, and to install IBM and Third party software. The templates are applied by using the built-in Terraform, Ansible, Helm, CloudPak, and Operator capabilities in Schematics.

As part of this getting started tutorial, you create a Schematics Workspaces that points to the [VSI database](https://cloud.ibm.com/catalogabout) template. Then, you run this template and watch Schematics provision your VSI and your IBM Cloud Databases for PostgreSQL instance. IBM Cloud Databases for PostgreSQL is a fully managed database offering in IBM Cloud that supports storing of non-relational and relational data types. For more information about this offering, see [What is PostgreSQL?](https://www.ibm.com/cloud/databases-for-postgresql).

This getting started tutorial incurs costs. You must have an [IBM Cloud Pay-As-You-Go or Subscription](https://cloud.ibm.com/registration) account to proceed. Make sure that you review pricing information for [classic VSIs](https://cloud.ibm.com/gen1/infrastructure/provision/vs) and [PostgreSQL](https://cloud.ibm.com/databases/databases-for-postgresql/create).



 Before you begin 

Before you can use this template, you must complete the following tasks.



* Make sure that you have the permissions to [create classic virtual servers](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-managing-device-access).
To create a workspace with Agent, refer to [ibmcloud schematics workspace new with Agent](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-agent-new) command.

{
"name": "<workspace_name>",
"type": [
"<terraform_version>"
],
"location": "<location>",
"description": "<workspace_description>",
"tags": [],
"template_repo": {
"url": "<github_source_repo_url>"
"branch": "master"
},
"template_data": [
{
"folder": ".",
"type": "<terraform_version>",
"variablestore":
{
"name": "<variable_name1>",
"value": "<variable_value1>",
"type": "string",
"secure": true
},
{
"name": "<variable_name2>",
"value": "<variable_value2>",
"type": "bool",
"secure": false
}
]
}
]
}



JSON file component description

 Parameter Description 

 workspace_name Enter a name for your workspace. The maximum length of character limit is set to 1 MB. For more information, see [Designing your workspace structure](https://cloud.ibm.com/docs/schematics?topic=schematics-workspaces-planstructure-workspace). 
 terraform_version The Terraform version that you want to use to run your Terraform code. To use Terraform version 0.12, enter terraform_v0.12, and similarly terraform_v0.13, and terraform_v0.14.
! !
Command options<-- <table> -->Schematics Workspaces upload flags| Flag | Required / Optional | Description || ---------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || --id or -i | Required | The unique identifier of the workspace where you want to upload your tape archive file (.tar). To find the ID of your workspace, run ibmcloud schematics workspace list command. || --file or -f | Required | Enter the full file path on your local machine where your .tar file is stored.
The tool displays a number of messages as it creates the project directory and adds a number of directories and files to it.It also runs npm install to install all the project dependencies, as specified in package.json. This process might take some time.
5. Before you can create a model, you need to make sure that your current working directory is the project top-level directory. To do this, enter the following command:

cd <project directory name>
6. To create a model, enter the following command:

apic create --type model

The tool prompts you for the name of the model.

? Enter the model name:
7. Enter a name for the model. In general, you can use any alphanumeric characters plus dashes and underscores in a model name. All the data sources that you added to the project are listed and the tool prompts you to select a data source to use:

? Select the data-source to attach item to: (Use arrow keys)
8. Select the data source that you want to use and press Enter. The tool prompts you to select the model's base class:

? Select model's base class (Use arrow keys)
Model
< PersistedModel
ACL
AccessToken
Application
Change
Checkpoint
(Move up and down to reveal more choices)

For more information about each option, see [Using built-in models](https://loopback.io/doc/en/lb3/Using-built-in-models.html).
9. Select a base class and press Enter. The tool asks whether you want to expose the model's REST API.



? Expose branch via the REST API? (Y/n)

If the model is exposed over REST, then all the standard create, read, update, and delete operations are available via REST endpoints.



10. Enter your selection and press Enter. If you chose to expose the model over REST, the tool prompts you for the plural form of the model name.



?
Incorrect column name specification results in an empty column, that is, the column seems to contain no data. To solve such a problem, use the automatic schema detection, reorder the columns, or omit some columns.

The SHOW TABLES statement provides you with an overview of the existing tables in your instance. This statement allows an optional search filter to limit the number of results:

SHOW TABLES LIKE 'cus'

It is not possible to use a different namespace than default.

To clean up catalog entries for unused data, use the DROP TABLE statement. This statement removes the table definition from the catalog without affecting the actual data on Object Storage:

DROP TABLE customers





 Partitioned tables 

You can manage a table in the catalog that references data that is organized in multiple partitions on Object Storage. The naming of the objects must adhere to the Hive-style partition naming convention: The object names must include the structure /columm=value/. The column must be a column name that is included in the schema definition of the CREATE TABLE statement. You can also have more than one partitioning columns in the object names, such as /columm1=value/column2=value/.

Following is an example list of object names on Object Storage that is partitioned on the country column with the Hive-style partition naming convention:

customers_partitioned.csv/country=Germany/cust-1.csv
customers_partitioned.csv/country=Germany/cust-2.csv
customers_partitioned.csv/country=Spain/cust-1.csv
customers_partitioned.csv/country=Austria/cust-1.csv
customers_partitioned.csv/country=Austria/cust-2.csv
customers_partitioned.csv/country=USA/cust-1.csv
customers_partitioned.csv/country=USA/cust-2.csv
customers_partitioned.csv/country=USA/cust-3.csv
customers_partitioned.csv/country=Sweden/cust-1.csv

To query partitioned tables, you must perform two mandatory steps:



 Step 1: Register the table
Creating custom dashboards in the Web UI 

When you create a custom dashboard, you can start from a template such as a pre-defined dashboard, or choose a blank dashboard. A dashboard includes panels that are configured to display specific data in a number of different formats. You also set how data is aggregated. The scope defines what data is used for aggregation and displayed. You can set the scope at a dashboard level, or override for individual panels.

Complete the following steps to create a custom dashboard:



1. Navigate to the Dashboards section in the Web UI
2. You can create a dashboard using a template or by creating a dashboard manually.



* To create a dashboard using a template:



1. Select a template from the DASHBOARD TEMPLATES list.
2. Click Create Custom Dashboard.
3. Name your dashboard.
4. Click Create and Open



* To create a dashboard without a template:



1. Click Add Dashboard![Add dashboard icon](https://cloud.ibm.com/docs-content/v1/content/220cdcbf6f4421bd7fee83fd1028048dba3f16c8/monitoring/monitor/images/add.png). The New Dashboard page opens.
2. Modify your dashboard as desired.
3. Click Save.
4. Click New Dashboard to rename your dashboard. Click the check mark to save your name change.





3. Set the dashboard scope. Click the Pencil icon ![Pencil icon](https://cloud.ibm.com/docs-content/v1/content/220cdcbf6f4421bd7fee83fd1028048dba3f16c8/monitoring/images/pencil.png). The select the desired scope. By default, Entire infrastructure is selected.



1. Select the scope.
2. Optionally, click Override the custom panel scopes to override the scope for all panels which currently have a custom scope defined.

This action cannot be undone.
For more information, see [FAQ](https://cloud.ibm.com/docs/schematics?topic=schematics-blueprints-faqfaqs-bp-target-region).
* Check that you have the [IAM permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-accessblueprint-permissions) to create blueprints.



Syntax to create blueprint config using CLI parameters:

ibmcloud schematics blueprint create --name BLUEPRINT_NAME [--resource-group RESOURCE_GROUP] [--description BLUEPRINT_DESCRIPTION] [--location BLUEPRINT_LOCATION] [--source-type BLUEPRINT_SOURCE_TYPE] --bp-git-url BLUEPRINT_GIT_URL [-—bp-git-file BLUEPRINT_GIT_FILE] [--bp-git-branch BLUEPRINT_GIT_BRANCH] [--bp-git-release BLUEPRINT_GIT_RELEASE] [--input-git-url INPUT_GIT_URL] [-—input-git-file INPUT_GIT_FILE] [--input-file BLUEPRINT_INPUT_FILE] [--input-git-branch INPUT_GIT_BRANCH] [--input-git-release INPUT_GIT_RELEASE] [--inputs BLUEPRINT_INPUT_LIST] [--bp-git-token BLUEPRINT_GIT_TOKEN] [--input-git-token BLUEPRINT_INPUT_GIT_TOKEN] | --file CONFIG_FILE_PATH [--output OUTPUT]

Command options



Schematics blueprint create flags

 Flag Required / Optional Description 

 --name or -n Required Name of the blueprint. Delimited by double quotes if the name contains spaces. For example -name "my blueprint" 
 --resource-group or -r Required The management resource group for the blueprint. 
 --description or --desc Optional The description of the blueprint. Delimited by double quotes if the description contains spaces.
* Make sure that you have the [permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-access) to create a workspace.



Ensure the location and the url endpoint are pointing to the same region when you list the Schematics Workspaces and actions. For more information about location and endpoint, see [Where is your information stored](https://cloud.ibm.com/docs/schematics?topic=schematics-secure-datapi-location)?



* Run a Schematics apply job against your workspace. An apply job provisions, modifies, or removes the IBM Cloud resources that you describe in the Terraform template that your workspace points to. Depending on the type and number of resources that you want to provision or modify, this process might take a few minutes, or even up to hours to complete.



During workspace plan execution, you cannot edit your workspace. After all updates are applied, the states of the files persist to determine what resources exist in your IBM Cloud account.





 Generate a workspace plan using the UI 



1. Log in to [IBM Cloud](https://cloud.ibm.com/).
2. Access Schematics > Workspaces > [workspace](https://cloud.ibm.com/schematics/workspaces).
3. Search your workspace in specific location.
4. Click Workspaces > [workspace](https://cloud.ibm.com/schematics/workspaces).
5. Click Apply plan to provision the configured resources.





 Verifying workspace plan 



1. Click your workspace that is listed in the [IBM Cloud console](https://cloud.ibm.com/schematics/workspaces) to view the results of the workspace details.









 Generate a workspace plan using the CLI 



1. Create a JSON file on your system and plan your workspace configuration.