vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Importing Schematics templates into the IBM Cloud catalog 

Understand how to [Create your private catalog](https://cloud.ibm.com/docs/account?topic=account-restrict-by-user), [manage your private catalog](https://cloud.ibm.com/docs/account?topic=account-filter-account), [assign access to the private catalog](https://cloud.ibm.com/docs/account?topic=account-catalog-access) in IBM Cloud®. And import your Terraform templates as products to make them available to your users. With a private catalog, you can limit the services that you want your users to see and the service settings that they can adjust. This way, you have more control over the type of service that is provisioned in your account and that naming conventions for services and service components are followed in your organization.



 Objectives 

In this tutorial, you import the IBM provided Observability Terraform template as a product to your private catalog to help users create the following IBM Cloud services at once:



* [IBM Cloud Log Analysis](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-getting-startedgetting-started) - Use this service to add logging capabilities to other IBM Cloud services, and to manage system and app logs.
* [IBM Cloud Monitoring](https://cloud.ibm.com/docs/monitoring?topic=monitoring-getting-started) - Use this service to gain operational visibility into the performance and health of your apps, services, and platforms.
* [IBM Cloud Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started) - Use this service to track any activity for a service so that you can comply with regulatory audit requirements.







 Time required 

30 minutes





 Audience 

This tutorial is intended for developers and system administrators who want to learn how to use Terraform templates to create and manage cloud infrastructure services by using IBM Cloud Schematics.





 Prerequisites 

Before you begin, make sure that you are assigned the following permissions:
Creating a database 

To create a database, submit a PUT request with the following format:

Method
: PUT /$DATABASE?partitioned=$BOOLEAN

Request body
: None

Response
: Success or failure of operation.

Roles permitted
: _admin

See the following example that uses HTTP to create a partitioned database:

PUT /$DATABASE?partitioned=true HTTP/1.1
HOST: $ACCOUNT.cloudant.com

See the following example that uses HTTP to create a nonpartitioned database:

PUT /$DATABASE?partitioned=false HTTP/1.1
HOST: $ACCOUNT.cloudant.com



 Example - Creating a partitioned database 

To create a partitioned database, see the following example:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X PUT "$SERVICE_URL/products?partitioned=true"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.Ok;
import com.ibm.cloud.cloudant.v1.model.PutDatabaseOptions;

Cloudant service = Cloudant.newInstance();

PutDatabaseOptions databaseOptions = new PutDatabaseOptions.Builder()
.db("products")
.partitioned(true)
.build();

Ok response =
service.putDatabase(databaseOptions).execute()
.getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.put_database(db='products', partitioned=True).get_result()

print(response)

putDatabaseOptions := service.NewPutDatabaseOptions(
"products",
)
putDatabaseOptions.SetPartitioned(true)
* [Reviewing Actions job details](https://cloud.ibm.com/docs/schematics?topic=schematics-action-workingaction-jobs)



[Creating an Ansible playbook](https://cloud.ibm.com/docs/schematics?topic=schematics-create-playbookcreate-playbook)



* [Referencing Ansible collections in your playbook](https://cloud.ibm.com/docs/schematics?topic=schematics-create-playbookschematics-collections)
* [Preparing your Ansible playbook to run in Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-create-playbookplan-ansible-playbook)



[Sample Ansible playbook templates for Schematics Actions](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatessample_actiontemplates)



* [Running cloud operations on Virtual Servers for VPC](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatesansible-vpc)
* [Provisioning a LAMP stack on Virtual Servers for VPC](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatesansible-lamp-stack)
* [Configuring IBM Cloud Databases for PostgreSQL with WAL2JSON plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatesansible-databases)
* [Automating app deployments in IBM Cloud Kubernetes Service](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatesansible-iks-deploy)
* [Installing kubectl on Virtual Servers for VPC](https://cloud.ibm.com/docs/schematics?topic=schematics-sample_actiontemplatesansible-kubectl)
For example, https://github.com/Cloud-Schematics/blueprint-basic-example/blob/main/basic-input.yaml. Review the [blueprint URL FAQ](https://cloud.ibm.com/docs/schematics?topic=schematics-blueprints-faqfaqs-bp-url) for details of the URL format.
* Source name - Used to set the source name the input file values are identified by in the UI.
* Personal access token - <Provide your Git personal access token, only for private Git repos>. For more information about how to create an access token, see [creating a personal access token for the command line](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
* Click Import values.





* Observe that the input variables from the inputs.yaml file are imported. Optionally, you can edit the variables. Enter variable values into the table by typing them in or by importing them. For the filled default values, if any, were pulled from the blueprint template, but can be changed. If there is a dropdown, select a value from the dropdown.
* Click Done editing, if the editing is done.
* Click Save draft only if you need to save the draft to continue edit the input variables again later.





4. Click Create Blueprint that will redirect to the blueprint overview page.







 Applying the blueprint configuration to create cloud resources 

You can follow these steps to generate a plan and apply a blueprint using the IBM Cloud console.



1. Log in to [IBM Cloud](https://cloud.ibm.com/).
2. Click Schematics > Blueprints.
3. Click your blueprint that is listed in the [Blueprints dashboard](https://cloud.ibm.com/schematics/blueprints)
4. Click Generate plan to initialize the modules that are defined in the blueprint template
The schemas of the objects do not need to be identical; depending on the input format, the schemas of multiple objects can be merged:



* For CSV format, columns are matched based on their order. Some input objects can contain more columns than others, but common columns must always use the same order across objects. The number of columns in the composite input is the maximum number of columns from all matched objects.
* For JSON and Parquet format, columns are matched based on their name. The set of columns in the composite input is the union of all column names from matched objects. For Parquet format, you must use the MERGE SCHEMA option of the [STORED AS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) clause to indicate that schema merging is to be performed.
* Input schema merging is not supported for AVRO and ORC formats. The first object determines the set of columns in the composite input, all columns that do not occur in the first object are ignored. The same behavior applies to Parquet, if the MERGE SCHEMA option is not specified.



Matching columns must have compatible data types across all objects where they appear. If a column does not appear in some of the input objects, it is padded with NULL values in the composite input.







 Object result set 

By default, the following three objects are written to Cloud Object Storage as a result set per job:



1. <target>/jobid=<job_id>
2. <target>/jobid=<job_id>/_SUCCESS
3. <target>/jobid=<job_id>/<part-number>



Only the last object contains the result set. The other two objects are empty and don't contain any data. It is important not to delete any of the objects if you want to use the result set for subsequent queries. By default, the object names include the job ID. For example, if you specify mydir/out or mydir/out/ as the target directory, the result objects are written under mydir/out/jobid=<job_id>.
[Launch the Knowledge Studio application](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutintrolaunching-the-knowledge-studio-application).
2. If you already have other workspaces, click Create workspace.
3. Click Create advanced rules workspace. Enter a name for your workspace, then click Create. The advanced rules workspace will load after a few seconds.



2. Create a new project (optional)

![New project button](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/create-advanced-rules-project.png)



1. In the Projects tab of your workspace, click the New button.
2. Enter a name for your project and click Create.



For more information, see [Managing projects and extractors](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractors).
3. Add documents to your project

![Add Documents button](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/advanced-rules-add-documents.png)



1. Click the Add Documents button.
2. Select the file type and language of your file.
3. Click Choose File and select a file to upload.
4. Click Add to add your file to the project.



4. Create an extractor



1. Follow the tutorials from the Get Help menu for more information on how to create and run extractors.
event-streams.schema.create A schema or schema version was created or updated in the Event Streams schema registry for the enterprise instance either through the administration API or through the Confluent Serdes. 
 event-streams.schema.delete A schema or schema version was deleted from the Event Streams schema registry for the enterprise instance. 
 event-streams.schema-rule.create A new rule or global rule was created in the Event Streams schema registry for the enterprise instance. 
 event-streams.schema-rule.update An existing rule or global rule was updated in the Event Streams schema registry for the enterprise instance. 
 event-streams.schema-rule.delete A rule was deleted in the Event Streams schema registry for the enterprise instance. 







 Where to view the events 

Activity Tracker events are available in the Activity Tracker account domain that is available in the IBM Cloud location (region) where the events are generated.

Events that are generated by an instance of the Event Streams service are automatically forwarded to the IBM Cloud Activity Tracker service instance that is available in the same location.

IBM Cloud Activity Tracker can have only one instance per location. To view events, you must access the web UI of the IBM Cloud Activity Tracker service in the same location where your service instance is available. For more information, see [Launch the web UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-startedgs_step4).





 How to enable message audit events 

Message audit events can be enabled on a per topic basis. To do so, complete the following steps:



1. Install Event Streams CLI plug-in v2.3 or later:

ibmcloud plugin install event-streams
2. Enable message audit on an existing topic:

ibmcloud es topic-update <topic-name> --config message.audit.enable=true

Or create a new topic with message audit enabled:

ibmcloud es topic-create <topic-name> --partitions <number-of-partitions> --config message.audit.enable=true
Make sure that your Terraform config files are compatible with the Terraform version that you specify. If the Terraform variable version is not specified, by default, Schematics selects the version from your template. 
 location Enter the location where you want to create your workspace. The location determines where your Schematics Actions run and where your workspace data is stored. The location is independent from the region where you want to create your IBM Cloud services. 
 description Enter a description for your workspace. 
 github_source_repo_url Enter the URL to the GitHub or GitLab repository where your Terraform configuration files are store 
 variable_name Optional, enter the name for the input variable that you declared in your Terraform configuration files. 
 variable_value Optional, enter the value for your input variable. 
 variable_type Optional, enter the data type of your input variable. For supported data types, see the [ibmcloud schematics workspace new command](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new). 



3. Verify that the workspace is created successfully.

curl -X GET https://schematics.cloud.ibm.com/v1/workspaces -H "Authorization: <iam_access_token>"
4. See [Managing IBM Cloud resources with Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) to create, update, or delete IBM Cloud resources with Terraform.







 Creating a workspace using a Terraform template 



1. Follow the steps in [Setting up Terraform for Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-terraform-setup) to create your workspace with Terraform.
2.
Go to your console's device menu and make sure that you have the correct account permissions to complete the tasks.
2. From the Devices menu, select Device List, and click the device that you want to use to create an image template.
3. Click the Actions dropdown and click Create image template. Make sure that the usernames and passwords that are listed match the operating system passwords and any other software add-on passwords. If passwords don't match, you can't create an image.
4. Enter a unique name for the image in the Image name field.
5. Enter any notes for the image in the Note field.
6. Select the Agree checkbox after you enter all the required information.
7. Click Create custom image to create the custom image.



Your server is offline until the imaging process is complete. The image template processing time varies based on the resources that are available on the physical host and how much data is being captured in the image template.





 Next steps 

After an image is created, more bare metal servers can be created by using the template. The new bare metal servers have the same configurations that are included in the image template. For more information, see [Ordering a bare metal server from an image template](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-ordering-bm-from-image-template).
See [Establishing a type system](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-typesystem) <br> <br>- Source documents: Create a corpus by uploading sample documents that are representative of your domain content into the workspace. See [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotation). Partition the corpus into document sets, specify the percentage of documents that are shared among all document sets, and assign the document sets to human annotators. <br> <br>- Dictionaries: Upload or create dictionaries for annotating text. You can choose to manually add dictionary entries or upload entries from a file, and then edit the entries. See [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-dictionaries). 
 Optional: Pre-annotate documents Pre-annotate documents according to the terms in the workspace dictionaries or based on rules that you define. See [Bootstrapping annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotation). 
 Annotate documents <br><br>1. Admins and project managers can annotate documents directly, without needing to create annotation tasks or go through the adjudication process. See [Annotating document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotating-document-sets-directly).<br>2. The project manager generally assigns annotation tasks to human annotators, configures the inter-annotator agreement threshold, and provides annotation guidelines for the human annotators to follow.
Step 6: (Optional) Enable the schema registry API 

The schema registry API is not automatically enabled when you provision an Event Streams Satellite instance. You must provide an IBM Cloud Object Storage bucket as the backend storage for schema registry to enable this API. You are responsible for managing this bucket, including but not limited to: data encryption, data backup, and disaster recovery.



1. Create a Cloud Object Storage instance either on IBM Cloud or on a Satellite location.

The Cloud Object Storage instance on Cloud can be in the same account or in a different account as the Event Streams Satellite instance. The Cloud Object Storage instance on Satellite must be in the same account and in the same location as the Event Streams Satellite instance.
2. Create a bucket in the Cloud Object Storage instance.

If the Cloud Object Storage instance is on IBM Cloud, ensure that the bucket is in the same region as the Event Streams Satellite instance's control region. If the Cloud Object Storage instance is on Satellite, ensure that the Cloud Object Storage instance's location is the same as the Event Streams Satellite instance's location.

To obtain Event Streams Satellite instance's control region or location, check its CRN, as in the following example.

crn:v1:bluemix:public:messagehub:satloc_dal_c9ntbe5f0gmsm06ofoq0:a/b5b95705e299425cb5c3c82e54d4533b:6b6e769a-f3c8-4e36-aa59-0736cdc036af::

Where satloc_dal_c9ntbe5f0gmsm06ofoq0 is the scope, dal is the control region's short name and indicates the us-south region, and c9ntbe5f0gmsm06ofoq0 is the location ID.
3. Create an authorization policy between the Event Streams Satellite instance and the Cloud Object Storage bucket.

The source is the Event Streams Satellite instance. The target is the Cloud Object Storage bucket, and the role is Writer.
For a sample syntax, see [File format](https://cloud.ibm.com/docs/schematics?topic=schematics-inventories-setupinv-file-format). Review the [limitations](https://cloud.ibm.com/docs/schematics?topic=schematics-inventories-setupinv-file-limitation) to ensure that your inventory definition is supported in Schematics.
5. Click Create inventory.
6. Follow the [steps](https://cloud.ibm.com/docs/schematics?topic=schematics-action-workingcreate-action) to create a Schematics Actions and use the resource inventory that you created.





 File format 

Review the following sample hosts.ini file to see the structure of the static host inventory list. For more information about hosts.ini file structures, see [Ansible documentation](https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.htmlhow-to-build-your-inventory).

 {: screen}
mail.example.com

[webservers]
web1.mydomain.com
web2.mydomain.com

[dbservers]
172.16.5.9
172.16.5.10
172.16.5.11

[datacenters]
172.15.4.0






 Limitations 

Review the following limitations of static inventory files in Schematics:



* Schematics supports only the INI file syntax.
* Variables are not supported in hosts.ini files.
* Specifying host groups by using key-value pairs in hosts.ini files is not supported.
* You must manually update the hosts.ini file if hostname or IP address of target hosts change.
* All target hosts must be configured with the same public SSH key. When you use the static inventory file in your Schematics action, you can specify one SSH key to authenticate with all target hosts that are included in your resource inventory.
Create a nickname for the database:
create nickname <nickname> for <server_name>.<schema_name>.<table_name>

For example:
create nickname ntest1 for db2server.admin2.testdata
4. Test that you can pull data from the target server:
select * from <nickname>

For example:
select * from ntest1









 Additional information 

For more information about data virtualization (federation), see: [Federation](https://www.ibm.com/support/knowledgecenter/SS6NHC/com.ibm.swg.im.dashdb.doc/fcontainer.html).

For information about the data sources supported by federation, see: [Federation Supported Data Sources](https://www.ibm.com/support/docview.wss?uid=swg27050561).
Select Start creating from Start from source code.
3. Select Application.
4. Enter a name for the application. Use a name for your application that is unique within the project.
5. Select a project from the list of available projects. You can also [create a new one](https://cloud.ibm.com/docs/codeengine?topic=codeengine-manage-projectcreate-a-project). Note that you must have a selected project to deploy an app.
6. Select Source code.
7. Click Specify build details.
8. Select a source repository, for example, https://github.com/IBM/CodeEngine. Because we are using sample source that does not require credentials, select None for the Code repo access. You can optionally provide a branch name. If you do not provide a branch name and you leave the field empty, Code Engine automatically uses the default branch of the specified repository. Click Next.
9. Select a strategy for your build and resources for your build. For more information about build options, see [Planning your build](https://cloud.ibm.com/docs/codeengine?topic=codeengine-plan-build). Click Next.
10. Select a container registry location, such as IBM Registry, Dallas to specify where to store the image of your build output. If your registry is private, you must [set up access](https://cloud.ibm.com/docs/codeengine?topic=codeengine-add-registry) to it.
11. Provide registry information about where to store the image of your build output. Select an existing Registry access secret or create a new one. If you are building your image to a Container Registry instance that is in your account, you can select Code Engine managed secret and let Code Engine create and manage the secret for you.
12. Select a namespace, name, and a tag for your image.
The Schematics ibmcloud terraform command usage displays warning and deprecation message as Alias 'terraform' will be deprecated. Use 'schematics' or 'sch' in your commands.



 Create file template in JSON format 

Schematics supports to download the Terraform modules template from the private repository. For more information, see [Supporting to download modules from private remote host](https://cloud.ibm.com/docs/schematics?topic=schematics-download-modules-pvt-git).

You can create the JSON file as shared in the example.json file for workspace creation and pass the file path along with the file name in --file flag. The description of all the parameters of example.json as described in the table.

You need to replace the <...> placeholders with the actual values. For example, "<workspace_name>" as "testworkspace".

example.json:

{
"name": "<workspace_name>",
"type": [
"<terraform_version>"
],
"location": "<location>",
"description": "<workspace_description>",
"tags": [],
"template_repo": {
"url": "<github_source_repo_url>"
},
"template_data": [
{
"folder": ".",
"type": "<terraform_version>",
"env_values":
{
"VAR1":"<val1>"
},
{
"VAR2":"<val2>"
}
],
"variablestore":
{
"name": "<variable_name_x>",
"value": "<variable_value_x>",
"type": "string",
"secure": true,
"description":"<description>"
},
{