vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Schematics Agents are a [beta feature](https://cloud.ibm.com/docs/schematics?topic=schematics-agent-beta-limitations) that are available for evaluation and testing purposes. It is not intended for production usage. Refer to the list of [limitations for Agents](https://cloud.ibm.com/docs/schematics?topic=schematics-agent-beta-limitations) in the beta release.



 Create profile_id for Agents 

Enable and configure your Agent service to establish trust with computed resources for Kubernetes cluster by using [trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profilecreate-profile-compute) as listed in the steps.



1. Describe your profile.



* Open the [IBM Cloud console](https://cloud.ibm.com).
* Click Manage > Access (IAM) > Trusted profiles. For more information about trusted profiles, see [Trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile&interface=ui).
* Click Create +.
* Name the profile myagent-profile.
* Click Continue.



2. Establish trust to the trusted profile.



* From the Trust relationship tab, select Compute resources.
* Select Compute service type as Kubernetes.
* Select Specific resources.
* Click Add a resource +.
* In Allow access to, select your cluster.
* In Namespace enter schematics-job-runtime.
* For Service account, enter default.
* Click Continue.



You created a trusted profile for a compute resource (Cluster running agent micro-services in schematics-job-runner namespace in your default service account.
3. Assign access to the trusted profile.



* Click the Access policy.
* Select Schematics service.
* Click Next.
* Select All resources.
* Click Next.
* Select Operator role.
* Click Add.
Creating Ansible roles and galaxy 

An [Ansible role](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html) can be used to separate one significant Ansible playbook into smaller reusable pieces called roles. A role defines a set of tasks that you want to run on your target hosts. To run these tasks on your hosts, you must reference the role in your Ansible playbook.

[Ansible Galaxy](https://galaxy.ansible.com/docs/?extIdCarryOver=true&sc_cid=701f2000001OH6uAAG) is a repository for Ansible roles that are available to drop directly into your Playbooks to streamline your automation projects. A new sysadmin might start automating with Ansible in a matter of a few hours.

You can [create your own roles](https://cloud.ibm.com/docs/schematics?topic=schematics-ansible-roles-galaxymain-file) or [use existing roles from Ansible Galaxy](https://cloud.ibm.com/docs/schematics?topic=schematics-ansible-roles-galaxyrequirements-file).



 Creating your own roles in Ansible 

To streamline your Ansible playbook, you can decide to separate out playbook tasks by creating roles and referencing them in your playbook. Use it to automatically load related variables, files, tasks, handlers, and other Ansible artifacts based on a known file structure.



1. Identify the tasks in your playbook that you want to reuse across multiple hosts. For example, you can group tasks that you want to run on all your hosts, and tasks that you want to run on your web servers and your databases. Each group of tasks can become its own role.
2. Create the Ansible role structure in your GitHub repository. Roles must be stored in a roles directory relative to your Ansible playbook. The roles directory can have a subdirectory such as /roles/db/ describing the tasks in the main.yml file.

├── roles
└── db
The schema information also allows developers who know AQL but not PMML to understand the AQL rule set.

The AQL compiler checks the input schema against the input schema of the PMML file to ensure that the two schemas are compatible. If the two schemas contain fields with the same name but incompatible types, compilation fails with an appropriate error message. If the function’s input or output schemas contain extra or missing columns, the resulting function ignores these columns and does not generate an error. The order of column names can be different between the AQL definition and the PMML file. If a column name appears in both input and output schemas but not in the PMML schema, then the values of that column are passed to the output of the function.





 Examples 

Example 1: Declaring scalar UDFs with scalar values as input using Java

The following example shows a create function statement that declares a function that is named udfCombineSpans. The user-defined function takes in two spans as input and returns a merged span similar to the first input span. The actual UDF Java™ function is packaged in a JAR file that is named udfs.jar, under the udfjars directory. The method that implements the function is combineSpans in class com.ibm.test.udfs.MiscScalarFunc. The method also declares a function udfSpanGreaterThan that returns true if the span is greater than the specified size.

/
* A function to combine two spans and return the resultant span from beginOffset of first span, to endOffset of second span
* @param p1 first input span
* @param p2 second input span
* @return a span from beginOffset of p1 till endOffset of p2
/

create function udfCombineSpans(p1 Span, p2 Span)
return Span like p1
external_name 'udfjars/udfs.jar:com.ibm.test.udfs.MiscScalarFunc!combineSpans'
language java
deterministic
return null on null input;

/
After selecting the appropriate settings, click Create to start the provisioning process. The Resource List page opens with the watsonx.data service that is listed under Databases.







 Step 2: Connecting to web console 

After creating the watsonx.data instance, you need to log in to the watsonx.data web console. Complete the following steps:



1. Log in to your IBM Cloud Account.
2. Go to Resource list>Databases.
3. Click your watsonx.data instance link. The service instance page opens.
4. Click the Open web console button to start the web console.
5. Log in to the console with your IBM ID and password.







 Step 3: Adding infrastructure 

When you log in to the watsonx.data web console for the first time, you are presented with the quick start wizard.

Use the quick start wizard to configure the basic settings and get started with watsonx.data in a few minutes.

In this section of the tutorial, you perform the following:



* Register the details of the object storage bucket that you bring in.
* Add the metadata catalogs (Iceberg catalog) to manage your table schemas.
* Select the query engine to work with your data.



Complete the following steps:



1. In the Bucket configuration page, select Register my own option and provide the following details to register your data bucket.



Table 1. Add bucket

 Field Description 

 Type Select from Amazon S3 or IBM Cloud Object Storage. 
 Name Enter your bucket name. 
 Endpoint Enter the endpoint URL. 
 Access key Enter your access key. 
 Secret key Enter your secret key. 



When you register your own bucket, ensure to provide the correct details for bucket configuration. Quick start wizard does not validate the bucket configuration details and you cannot modify them later.

Ensure that the data bucket contains data.
2. Click Next. This displays the Catalogs configuration page.
3. In the Catalogs configuration page, select Apache Iceberg.
4. Click Next. This displays the Engine configuration page.
5.
For more information, see [Designing your workspace structure](https://cloud.ibm.com/docs/schematics?topic=schematics-workspaces-planstructure-workspace). 
 terraform_version The Terraform version that you want to use to run your Terraform code. To use Terraform version 0.12, enter terraform_v0.12, and similarly terraform_v0.13, and terraform_v0.14. Make sure that your Terraform config files are compatible with the Terraform version that you specify. If the Terraform variable version is not specified. by default, Schematics selects the version from your template. 
 location Enter the location where you want to create your workspace. The location determines where your Schematics Actions run and where your workspace data is stored. The location is independent from the region where you want to create your IBM Cloud services. 
 description Enter a description for your workspace. 
 github_source_repo_url Enter the URL to the GitHub or GitLab repository where your Terraform configuration files are stored. If you choose to create your workspace without a GitHub repository, your workspace is created with a draft state. To connect your workspace to a GitHub repository later, you must use the ibmcloud schematics workspace update command. 
 variable_name Optional, enter the name for the input variable that you declared in your Terraform configuration files. 
 variable_type Optional, enter the data type of your input variable. For supported data types, see the [ibmcloud schematics workspace new command](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new). 



2. Create the workspace.

ibmcloud schematics workspace new --file workspace.json
3. Verify that your workspace is created. Make sure that your workspace is in an Inactive state.

ibmcloud schematics workspace list
4.
* Make sure that you have the required [permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-accessaccess-setup) to [create VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started), to create an endpoint gateway, to create or bind a reserved IP from the subnet, and [account limits for VSI creation](https://cloud.ibm.com/docs/vpc?topic=vpc-quotasvpcquotas) for concurrent instances.
* A VPC Generation 2 instance and a subnet zones to bind an IP address at the same time you provision the endpoint gateway. For more information, see [Getting started with VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-a-vpc-using-the-ibm-cloud-console).
* A VSI is created. For more information, see [creating a VSI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-virtual-servers&interface=cli).







 Adding Virtual Private Endpoint Gateways for Schematics 

Now, you can securely connect the Virtual Private Endpoint Gateways to access Schematics services and functions such as workspace, action, job, plan, apply, and destroy for a new instance. For more information, see [Overview of private service endpoints in Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-secure-datapi-location).

You cannot create multiple Virtual Private Endpoint Gateways for the same Schematics instance.

The steps to add the private network endpoints for Schematics:



1. Create a Schematics Workspace. For more information, see [creating a workspace](https://cloud.ibm.com/docs/schematics?topic=schematics-sch-create-wks&interface=uicreate-wks-ui).
2.
Schematics Workspaces update flags

 Flag Required / Optional Description 

 --id or -i Required The unique identifier of the workspace for which you want to update the instance or resource. To find the ID of your workspace, run ibmcloud schematics workspace list command. 
 --file or -f Optional The relative path to a JSON file on your local machine that includes the updated parameters for your workspace. For more information about the sample JSON file with the details, see [JSON file update template](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referencejson-file-update-template). 
 --github-token or -g Optional Enter the GitHub token value to access private Git repository. 
 --pull-latest or --pl Optional Pull the latest changes from your GitHub repository into workspace. If this flag is set --file flag is ignored. 
 --output or -o Optional Returns the command-line output in JSON format. Currently only JSON file format is supported. 
 --json or -j Deprecated Prints the output in the JSON format. 





 Update file template in JSON format 

You can create the JSON as shared in the example.json file for workspace update and pass the file path along with the file name in --file flag. The description of all the parameters of example.json is described in the table.

You need to replace the <...> placeholders with the actual values. For example, "<workspace_name>" as "testworkspace".

example.json:

{
"name": "<workspace_name>",
"type": "<terraform_version>",
"description": "<workspace_description>",
"tags": [],
"resource_group": "<resource_group>",
"workspace_status": {
"frozen": "<true_or_false>"
},
"template_repo": {
If you are not familiar with Ansible, you can use one of the [IBM provided playbooks](https://github.com/Cloud-Schematics), or browse the [Ansible Galaxy library](https://galaxy.ansible.com/).
2. Create a Schematics action: When you create an action, you point your action to the repository that stores your Ansible playbook. Then, you select the cloud resources where you want to run the tasks that are defined in your Ansible playbook. To protect your cloud resources, you can further set up a bastion host in front of your target hosts that proxies all Ansible SSH connections to the target hosts.
3. Run your action: When you are ready to configure your cloud resources, you can run your action. Schematics uses the built-in Ansible engine to connect to your target hosts via SSH, and execute the tasks that are defined in your Ansible playbook. You can monitor the progress by reviewing the logs.







 Next steps 

So far you have learned about Schematics Actions. The following are some next steps to explore.



* See [Creating Actions](https://cloud.ibm.com/docs/schematics?topic=schematics-action-workingcreate-action) to dig into how to create an action?
Stopped The Schematics plan, apply, or destroy job are stopped manually. 
 Template_Error The Schematics template contains errors and cannot be processed. 







 Workspace state diagram and manipulative job 

The state of a workspace indicates if you have successfully created a Terraform execution plan and applied to provision your resources in the IBM Cloud account. The table represents the state and the workspace job.



Workspace state diagram

 workspace State diagram Description 

 Create workspace ![Create workspace state](https://cloud.ibm.com/docs-content/v1/content/ba89e173287b28d1453d2a327d8a1f74ae2f1662/schematics/images/createworkspace.png) The workspace is created without a reference to GitHub, GitLab, or Bitbucket to the draft state. From the draft state you can connect to the infrastructure template in your source repository. From connecting state, the template is processed successfully to reach Inactive state (Final state) or template parsing may fail and reach failed state. From inactive state, when you do an apply, and if it results in one resource then, state enters active state and if they destroy, state enters destroy state. you can maintain at least one resource in the state file by apply job, to move the workspace into active state. The Schematics [stores](https://cloud.ibm.com/docs/schematics?topic=schematics-general-faqpersist-file) the user-defined file for running the subsequent Terraform commands. Then, you can destroy all the resources to make your workspace in an inactive state. 
 Delete workspace ![Delete workspace state](https://cloud.ibm.com/docs-content/v1/content/ba89e173287b28d1453d2a327d8a1f74ae2f1662/schematics/images/deleteworkspace.png) When you perform delete workspace on an inactive, active or failed state.
After you enable VRF and service endpoints for your account, all existing and future Schematics Workspaces become available from both the public and private service endpoints.
3. Verify that your account is enabled for VRF and service endpoints.



1. Log in to IBM Cloud.

ibmcloud login

If the login fails, run the ibmcloud login --sso command to try again. The --sso parameter is required when you log in with a federated ID. If this option is used, go to the link listed in the command-line output to generate a one-time passcode.
2. Show the details of your account.

ibmcloud account show

Example output:

Retrieving account User's Account of user@email.com...
OK

Account ID: a111aaaa1aa1aaaaaaaaaaaa1a1aa111
Currently Targeted Account: true
Linked Softlayer Account: 000000
VRF Enabled: true
Service Endpoint Enabled: true









 Connect to the Schematics private service endpoint 

Prepare your VSI or test machine by configuring your routing table for the IBM Cloud Private network.



1. To connect to the private service endpoint, you must create a virtual server instance (VSI) first. You use this VSI to connect to the IBM Cloud Private network. You can create a [classic VSI](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-getting-started-tutorial) or [VPC VSI](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started).
2. After you are connected to the VSI, target the private service endpoint when you send API requests to the Schematics API server. The following example shows the supported Terraform and Helm versions of the Schematics engine.

curl -X GET https://private-us-south.schematics.cloud.ibm.com/v1/version









 Virtual Private Endpoints Gateways for Schematics
Use the following CLI command to create a workspace with your config.json file. Make sure that the config.json file exists in the directory where you run the command.

ibmcloud schematics workspace new -f hpc_workspace_config.json --github-token GITHUB_TOKEN

The --github-token parameter is optional and only needed if you are using a private GitHub repository. If you are using the [IBM Cloud public GitHub repository](https://github.com/IBM-Cloud/hpc-cluster-openshift), you do not need to provide it.



 Listing available workspaces 

You can list the workspaces in your account by using the following command:

ibmcloud schematics workspace list

Example response with workspace details:

Name           ID       Description           Status         Frozen
oc-for-hpc    us-east.workspace.bcc-oc-test.7cbc3f6b       Sample workspace      INACTIVE   False





 Retrieving workspace details 

You can retrieve the details of an existing workspace, including the values of all input variables, by running the following command:

ibmcloud schematics workspace get --id WORKSPACE_ID [--output OUTPUT]





 Updating a workspace 

You can update the details for an existing workspace, such as the workspace name, variables, or source control URL by running the following command:

ibmcloud schematics workspace update --id WORKSPACE_ID --file FILE_NAME [--github-token GITHUB_TOKEN]

To provision or modify IBM Cloud resources, you can run the command ibmcloud schematics plan command. For more information, see the [IBM Cloud Schematics CLI](https://cloud.ibm.com/docs/hpc-openshift?topic=schematics-schematics-cli-referenceschematics-plan) reference.







 Next steps
You can create, update, and delete an agent assignment policy by using the Schematics [policy commands](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-policy-create) CLI.

The agent-assignment-policy for an agent is defined by using the following attributes of a workspace or an action. The selection attributes can be a combination of the following flags:



* tags – workspaces or actions with matching user tags are selected.
* locations – workspaces or actions in the matching Schematics location are selected.
* resource-groups - workspaces or actions with the matching resource-group are selected.



If the selection policy for agent-1 specified tags=[dev] and resource-group=[rg-2], Schematics automatically routes workspace jobs including Git download, Terraform plan, apply, destroy jobs for all workspaces that match the tags, and resource-group criteria, to execute on agent-1.



 Creating an agent policy using the CLI 

Create your agent policy using the CLI. For the complete list of agent policy options, see the [policy commands](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-policy-create) doc.

Before you begin:



* Install or update the [Schematics plug-in](https://cloud.ibm.com/docs/schematics?topic=schematics-setup-cliinstall-schematics-plugin) version to be 1.12.9 or higher.
* Select the IBM Cloud region where the agent is defined. Set the CLI region by running [ibmcloud target -r <region>](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_cliibmcloud_target) command.
* Check that you have the [IAM permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-accessblueprint-permissions) to create an agent policy.
Schematics Blueprints is a [beta feature](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-beta-limitations) that are available for evaluation and testing purposes. It is not intended for production usage. Refer to the list of [limitations](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-beta-limitationssc-bp-beta-limitation) for the beta release.







 Blueprint permissions 

Review the following table to see what permissions you need to work with Schematics Blueprint.

In addition to the listed blueprints activities and permission, you must check whether you have related [workspace permissions](https://cloud.ibm.com/docs/schematics?topic=schematics-accessworkspace-permissions) for blueprint create, blueprint apply, blueprint delete, and blueprint destroy activities to execute successfully.



User permissions for Schematics Blueprint
The table shows user permissions by access role. Rows are to be read from the left to right, with the access role in column one, and the permission descriptions in column two.

 Activities Reader Writer Manager Account owner 

 View blueprint ![Check mark icon.](https://cloud.ibm.com/docs-content/v1/content/ba89e173287b28d1453d2a327d8a1f74ae2f1662/schematics/images/checkmark.svg) ![Check mark icon.](https://cloud.ibm.com/docs-content/v1/content/ba89e173287b28d1453d2a327d8a1f74ae2f1662/schematics/images/checkmark.svg) ![Check mark icon.]
After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-applying-plan&interface=cli) to deploy your IBM Cloud resources to build a Slurm cluster instance.





 Generating a plan using the API 



1. To generate a plan by using IBM Cloud Schematics Python APIs, create a Python file and provide a name of your choice, for example, schematics_generate_plan.py.
2. Copy and paste the [Generate a plan using Schematics Python API](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-generate-plan&interface=apiexample-request-generate) example request to your Python file.
3. Change the following parameters as part of the request:



* Replace your IBM Cloud API key to the authenticator = IAMAuthenticator('<ibm-api-key>') variable.
* Change the API endpoint to the endpoint mentioned in [API endpoints](https://cloud.ibm.com/apidocs/schematics?code=pythonapi-endpoints) according to the location you want your Schematics workspace to reside, for example, schematics_service.set_service_url('https://us.schematics.cloud.ibm.com').



4. Inside the schematics_service.plan_workspace_command function, provide the following parameters:



* Provide the workspace ID that you created in the [Create a workspace](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-creating-workspace&interface=api) task or from the IBM Cloud console for Schematics, for example, us-south.workspace.Terraform-Schematics-Python-Workspace.b3bbc9f5.
* Run the following curl command to create a refresh token: