vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Change the API endpoint to the endpoint mentioned in [API endpoints](https://cloud.ibm.com/apidocs/schematics?code=pythonapi-endpoints) according to the location that you want your Schematics workspace to reside, for example, schematics_service.set_service_url('https://us.schematics.cloud.ibm.com').



4. Inside the schematics_service.apply_workspace_command function, provide the following parameters:



* Provide the workspace ID that you generated in the [Creating a workspace](https://cloud.ibm.com/docs/ibm-spectrum-lsf?topic=ibm-spectrum-lsf-creating-workspace) task, for example, us-south.workspace.Terraform-Schematics-Python-Workspace.b3bbc9f5.
* Export your IBM Cloud API key by using the following command:



export IBMCLOUD_API_KEY =”<ibm-cloud-api-key>”



* Run the following curl command to create a refresh token:



curl -X POST "https://iam.cloud.ibm.com/identity/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=$IBMCLOUD_API_KEY" -u bx:bx
5. Run the Python script by using python3 <python-file-name> to apply a plan in the IBM Cloud.
6. You get an activity ID in response if the parameters passed as part of the request are valid. You should see the plan being applied in the Schematics workspace that you created in the IBM Cloud console. If you don’t get a successful response, the error response contains the errors that you need to resolve. Resolve those errors and run the script until you are able to get a valid response and apply a plan.
7.
Make sure to update the required parameters, such as api_key, region, ssh_key_name, cluster_prefix, and worker_nodes_per_zone in the config.json file.
6. Run the Python script by using python3 to update the variables in the Schematics workspace in the IBM Cloud.



The following parameters might not be required in config.json as the Schematics update variables API uses the workspace ID w_id to update the variables against that workspace.

{
  "name": "Schematic Dev Workspace",
  "type": [
    "terraform_v0.13.7"
  ],
  "location": "us-south",
  "description": "Schematic Dev Workspace",
  "tags": [],
  "template_repo": {
    "url": "<GitHub repo URL>",
    "githubtoken": "<github-token>"
  }



 Example Python request for schematics_variables_update.py file 

The following Python example request is for the example file, schematics_variables_update.py.

import logging, os, json

logging.basicConfig()
logging.root.setLevel(logging.NOTSET)
logging.basicConfig(level=logging.NOTSET)

from schematics_env_class import HPCCEnvironmentValues

logging.info("Schematic Variable Update Started")

if __name__ == '__main__':

    json_files = os.path.join(os.path.abspath("."), "config.json")

    with open(json_files, "r") as file:
        config_data = json.load(file)

    api_key = config_data["template_data"]["variablestore"]["value"]

    schematic_obj = HPCCEnvironmentValues(api_key)
Retrieving action logs with Schematics API 



1. To retrieve the logs for an action, such as generate a plan, apply a plan, and destroy resources using IBM Cloud® Schematics Python APIs, create a Python file and provide a name of your choice, for example, schematics_logs_api.py.
2. Copy and paste the [Retrieve Schematics workspace action details using Python API](https://cloud.ibm.com/docs/hpc-spectrum-symphony?topic=hpc-spectrum-symphony-retrieve-action-logs&interface=apiexample-request-retrieve-action-logs) example request into your Python file.
3. Change the following parameters as part of the request:



* Replace the authenticator = IAMAuthenticator('<ibm-api-key>') variable with your IBM Cloud API key.
* Change the API endpoint to the endpoint mentioned in this link according to the location where you want your Schematics workspace to reside, for example, schematics_service.set_service_url('https://us.schematics.cloud.ibm.com').



4. Inside the schematics_service.get_workspace_activity function, provide the following parameters:



* Provide the workspace ID that you generated in the [Creating a workspace](https://cloud.ibm.com/docs/hpc-spectrum-symphony?topic=hpc-spectrum-symphony-creating-workspace&interface=api) task, for example, us-south.workspace.Terraform-Schematics-Python-Workspace.b3bbc9f5.
* Provide an activity ID that was generated as a successful response for the specific action for which you want to retrieve logs. You received an activity ID when you successfully completed the action in the Schematics workspace.



5. Run the Python script by using python3 <python-file-name> to retrieve the logs for an action for your Schematics workspace in the IBM Cloud.
6.
created virtual environment CPython3.9.10.final.0-64 in 3291ms
creator CPython3Posix(dest=/tmp/virtualenv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Collecting pyjokes
Downloading pyjokes-0.6.0-py2.py3-none-any.whl (26 kB)
Installing collected packages: pyjokes
Successfully installed pyjokes-0.6.0
6. Save the following code as __main__.py in your test directory. When you create actions with a compressed file, the source file that contains the entry point must be named __main__.py.

import pyjokes

def joke(params):
return {"joke": pyjokes.get_joke()}
7. To deploy this code as an action, you must create a compressed file of the virtualenv folder and the __main__.py file.

Sometimes, the resulting compressed file is larger than the maximum codeSize as described in the [Action Limits](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-limitslimits_actions) allowed by Cloud Functions. To reduce the size of the compressed file, select only the dependencies that you need, rather than selecting the entire virtualenv folder. The packages that you need can be found in the site-packages directory within the virtualenv folder.
console.log(Retrieving HEAD for item: ${itemName} from bucket: ${bucketName});
console.log(JSON.stringify(data, null, 4));
})
.catch((e) => {
console.log(ERROR: ${e.code} - ${e.message}n);
});
}

SDK References



* [headObject](https://ibm.github.io/ibm-cos-sdk-js/AWS/S3.html)









 Updating Metadata 

There are two ways to update the metadata on an existing object:



* A PUT request with the new metadata and the original object contents
* Executing a COPY request with the new metadata specifying the original object as the copy source





 Using PUT to update metadata 

Note: The PUT request overwrites the existing contents of the object so it must first be downloaded and re-uploaded with the new metadata.

function updateMetadataPut(bucketName, itemName, metaValue) {
console.log(Updating metadata for item: ${itemName});

//retrieve the existing item to reload the contents
return cos.getObject({
Bucket: bucketName,
Key: itemName
}).promise()
.then((data) => {
//set the new metadata
var newMetadata = {
newkey: metaValue
};

return cos.putObject({
Bucket: bucketName,
Key: itemName,
Body: data.Body,
Metadata: newMetadata
}).promise()
.then(() => {
console.log(Updated metadata for item: ${itemName} from bucket: ${bucketName});
})
})
.catch((e) => {
console.log(ERROR: ${e.code} - ${e.message}n);
});
}
Show more





 Using COPY to update metadata
* Change the API endpoint to the endpoint mentioned in [API endpoints](https://cloud.ibm.com/apidocs/schematics?code=pythonapi-endpoints) according to the location that you want your Schematics workspace to reside, for example, schematics_service.set_service_url('https://us.schematics.cloud.ibm.com').
* Provide environment values that you want to have in the form of a key and value pair, such as [{ 'Environment': 'hpc-dev-cluster' }]
* Change the folder location variable template_source_data_request_model['folder'] to the folder name inside your GitHub repository where your Terraform files reside. If the Terraform files exist in the base location of your repository, then set this variable to template_source_data_request_model['folder'] = ' '; otherwise, mention the folder name in the template_source_data_request_model['folder'] variable.
* Change the template_source_data_request_model['type'] variable to the Terraform version that you are using to create IBM Cloud resources such as terraform_v0.13. Ensure that your Terraform templates are compatible with the version that you are mentioning in this field.
* Provide your GitHub or GitLab Repository HTTPS URL where your Terraform files reside in the template_repo_request_model['url'] variable. If you are using the [public repository](https://github.com/IBM-Cloud/hpc-cluster-lsf) that is provided by IBM Cloud, then set this variable as template_repo_request_model['url'] = 'https://github.com/IBM-Cloud/hpc-cluster-lsf'; otherwise, set it to the private repository you are using.



4. Inside the schematics_service.create_workspace function, provide the following parameters:



* Provide an optional description.
* Provide a name to identify your Schematics workspace, for example, terraform-dev-workspace.
Upgrade your action code to use this new package, since the former does not receive updates. This package includes support for promises. 
 ibmiotf The package ibmiotf has been renamed by the maintainers to @wiotp/sdk. Make sure to update your action code to the new package. See [@wiotp/sdk](https://www.npmjs.com/package/@wiotp/sdk) for all changes. The package ibmiotf does not receive further updates and is planned to be removed from this runtime in the future. 



For more information about migrating to Node.js:10, see [(Details on GitHub)](https://github.com/ibm-functions/runtime-nodejs/blob/master/nodejs10/CHANGELOG.md).





 Node.js packages 

Node.js 12 packages

Node.js 10 packages



Table 1. Node.js 12 packages.

 Node.js 12 packages Description 

 [@cloudant/cloudant](https://www.npmjs.com/package/@cloudant/cloudant) This library is the official Cloudant library for Node.js. 
 [@ibm-functions/iam-token-manager](https://www.npmjs.com/package/@ibm-functions/iam-token-manager) This package is an IAM access token manager library for Node.js. 
 [@sendgrid/mail](https://www.npmjs.com/package/@sendgrid/mail) Provides email support via the SendGrid API. 
 [@wiotp/sdk](https://www.npmjs.com/package/@wiotp/sdk) IBM Watson IoT Platform JavaScript SDK. 
 [amqplib](https://www.npmjs.com/package/amqplib) A library for making AMQP 0-9-1 clients for Node.js.
pip3 install git+https://github.com/ibm-functions/composer-python.git
2. Confirm that you installed the library by running help for the Composer commands.

pycompose -h
pydeploy -h

Example output

usage: pycompose composition.py command [flags]

usage: pydeploy composition composition.json [flags]









 Configuring and running compositions in Cloud Functions 

You can use the JavaScript or Python Composer libraries to create your compositions in Cloud Functions. Use compose or pycompose to compile your composition source code, then use deploy or pydeploy to deploy the composition to Cloud Functions. After you configure the composition, you can run it in Cloud Functions.

Before you begin



1. Log in to IBM Cloud and install the [IBM Cloud and the Cloud Functions CLI](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-cli_install).
2. [Set up and target a namespace](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-namespacescreate_iam_namespace).



To run a composition, follow these steps



1. Create Composer source code with the Node.js or Python libraries. You can find some example code in the samples folder of each repository ([JS](https://github.com/ibm-functions/composer/tree/master/samples), [Python](https://github.com/ibm-functions/composer-python/tree/master/samples)).
2. Compile the Composer source code into a JSON file.



* JavaScript

compose demo.js > demo.json
* Python

pycompose demo.py > demo.json



3. Deploy the code to Cloud Functions. Include the -w flag to overwrite any existing deployment that is named demo.



* JavaScript

deploy demo demo.json -w
* Python
](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5)Cloud Databases v5 API Service SDK[Cloudant](https://cloud.ibm.com/apidocs/cloudant)Cloudant Service SDK[Data Engine (previously SQL Query) V3](https://cloud.ibm.com/apidocs/sql-query/sql-query-v3)Data Engine (previously SQL Query) V3 Service[Data Management Console](https://cloud.ibm.com/apidocs/data-management-console)Data Management Console Software[Db2 Warehouse on Cloud v4](https://cloud.ibm.com/apidocs/db2-warehouse-on-cloud/db2-warehouse-on-cloud-v4)Db2 Warehouse on Cloud v4 Service[Db2 Warehouse on Cloud v5 (AWS only)](https://cloud.ibm.com/apidocs/db2-warehouse-on-cloud/db2-warehouse-on-cloud-v5)Db2 Warehouse on Cloud v5 (AWS only) Service
* [Configuring the mobile SDK](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_mobile_sdkconfigure-mobile-sdk)



* [Support for qualified names with mobile SDK](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_mobile_sdkqualified-names-mobile-sdk)
* [Invoking actions with mobile SDK from WhiskButton](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_mobile_sdkinvoke-mobile-sdk-actions-whiskbutton)





[Object Storage](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage)



* [Packages for Object Storage](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstorageobstorage_packages)
* [Setting up the IBM Cloud Object Storage trigger](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage_ev)



* [Prerequisites for working with the IBM Cloud Object Storage trigger](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragecos_changes_pre)
* [1. Assigning the Notifications Manager role to your Cloud Functions namespace](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage_auth)
* [2.
} else {
var result = reply["result"]
print("Got result (result)")
}
})
} catch {
print("Error (error)")
}

By default, the SDK returns only the activation ID and any result that is produced by the invoked action. To get metadata of the entire response object, which includes the HTTP response status code, use the following setting:

whisk.verboseReplies = true





 Configuring the mobile SDK 

You can configure the SDK to work with different installations of Cloud Functions by using the baseURL parameter. For instance:

whisk.baseURL = "http://localhost:8080"

In this example, you use an installation that is running at http://localhost:8080. If you do not specify the baseURL, the mobile SDK uses the instance that is running at [https://us-south.functions.cloud.ibm.com](https://us-south.functions.cloud.ibm.com).

You can pass in a custom NSURLSession in case you require special network handling. For example, you might have your own Cloud Functions installation that uses self-signed certificates:

// create a network delegate that trusts everything
class NetworkUtilsDelegate: NSObject, NSURLSessionDelegate {
func URLSession(session: NSURLSession, didReceiveChallenge challenge: NSURLAuthenticationChallenge, completionHandler: (NSURLSessionAuthChallengeDisposition, NSURLCredential?) -> Void) {
completionHandler(NSURLSessionAuthChallengeDisposition.UseCredential, NSURLCredential(forTrust: challenge.protectionSpace.serverTrust!))
}
}
// create an NSURLSession that uses the trusting delegate
[External link icon](https://cloud.ibm.com/docs-content/v1/content/242c92ec4bec2bd4597959c2a4cd020918ee0c71/icons/launch-glyph.svg)](https://sldn.softlayer.com/reference/services/SoftLayer_Virtual_Guest_Block_Device_Template_Group/getPublicImages/)



For API usage examples, see the following resources:



* [How to create a virtual server from an image template with the SoftLayer API ![External link icon](https://cloud.ibm.com/docs-content/v1/content/242c92ec4bec2bd4597959c2a4cd020918ee0c71/icons/launch-glyph.svg)](https://stackoverflow.com/questions/41138874/how-to-create-virtual-server-using-standard-template-softlayer-using-rest-api)
* [SoftLayer API Python Client: Working with Virtual Servers ![External link icon](https://cloud.ibm.com/docs-content/v1/content/242c92ec4bec2bd4597959c2a4cd020918ee0c71/icons/launch-glyph.svg)](https://softlayer-api-python-client.readthedocs.io/en/latest/api/managers/vs/)
* [SoftLayer API Python Client: SoftLayer.image ![External link icon](https://cloud.ibm.com/docs-content/v1/content/242c92ec4bec2bd4597959c2a4cd020918ee0c71/icons/launch-glyph.svg)](https://softlayer-api-python-client.readthedocs.io/en/latest/api/managers/image/)
For example: value-"sdk-fs:///TMP/MY_TEST_DATA/"

The URLs must start with sdk-fs://. So to crawl, for example, /home/watson/mydocs, the value of this URL is sdk-fs:///home/watson/mydocs. The third / in the URL is necessary.

Save and close the file.
2. discovery_service.conf - Open the discovery/discovery_service.conf file in a text editor. Modify the following values specific to the Discovery instance you previously created on IBM Cloud:



* environment_id - Your Discovery environment ID.
* collection_id - Your Discovery collection ID.
* configuration_id - Your Discovery configuration ID.
* configuration - The full path location of this discovery_service.conf file, for example, /home/config/discovery/discovery_service.conf.
* apikey - Your Discovery credential.



3. crawler.conf - Open the config/crawler.conf file in a text editor.



* Set the output_adapterclass and config options for Discovery as follows:

class - "com.ibm.watson.crawler.discoveryserviceoutputadapter.DiscoveryServiceOutputAdapter"

config - "discovery_service"

discovery_service {
include "discovery/discovery_service.conf"
}



4. After modifying these files, you are ready to crawl your data.







 Crawl your data 

Run the following command: crawler crawl --config [config/crawler.conf]

This command runs a crawl with the configuration file crawler.conf.

The path to the configuration file passed in the --config option must be a qualified path. That is, it must be in relative formats, such as config/crawler.conf or ./crawler.conf, or in an absolute path, such as /path/to/config/crawler.conf.





 Search your documents
Two sample GitHub repositories are provided for you to test the GREP11 API:



* [The sample GitHub repository for Golang](https://github.com/IBM-Cloud/hpcs-grep11-go)
* [The sample GitHub repository for JavaScript](https://github.com/IBM-Cloud/hpcs-grep11-js)



You can use the following Golang code example to generate random data by calling the GenerateRandom function.

This example assumes that extra required Golang packages are included through import statements, such as the [gRPC](https://godoc.org/google.golang.org/grpc) and [http](https://golang.org/pkg/net/http/) packages. The import pb "github.com/IBM-Cloud/hpcs-grep11-go/grpc" statement is used by GREP11 to perform API function calls.

import pb "github.com/IBM-Cloud/hpcs-grep11-go/grpc"

// Data structure and supporting methods used for GREP11 authentication
// IAMPerRPCCredentials type defines the fields required for IBM Cloud IAM authentication
// This type implements the gRPC PerRPCCredentials interface

type IAMPerRPCCredentials struct {
expiration time.Time
updateLock sync.Mutex
AccessToken string // Required if APIKey nor Endpoint are specified - IBM Cloud IAM access token
APIKey string // Required if AccessToken is not specified - IBM Cloud API key
Endpoint string // Required if AccessToken is not specified - IBM Cloud IAM endpoint
}

// GetRequestMetadata is used by GRPC for authentication
func (cr IAMPerRPCCredentials) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
// Set token if empty or Set token if expired
* [Visual Studio](https://visualstudio.microsoft.com/downloads/)
* [Python v2.7.10](https://www.python.org/downloads/release/python-2710/)
* [Microsoft Windows SDK for Windows 10](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk)
* npm version 3: See the following note.



Note: Use Visual Studio Community unless you want to purchase Visual Studio Enterprise. Run the installer, check "Visual C++" under "Programming Languages", and accept the default installation location.

For 64-bit builds of Node.js and native modules, you also need the Windows™ 7 64-bit SDK. If the installation fails, try uninstalling any C++ 2010 x64&x86 redistributable that you have installed first. If you get errors that the 64-bit compilers are not installed, you might also need the compiler update for the Windows™ SDK 7.1.

Enter the following command to install npm version 3: npm install -g npm

Then ensure that the npm command uses the correct version: npm -v

If the version shown is not 3.x.x, then edit your system PATH to ensure that C:UsersusernameAppDataRoamingnpm supersedes any other entries.



1. Optional: If you are using the API Designer, open a new shell window.
2. When you have installed the compiler tools that are required for your operating system, install the connector by entering the following command in the project root directory.

npm install --save <connector-package>



where <connector-package> is the name of the npm package for the LoopBack connector, as shown in the Table 1:



Table 1. LoopBack Connectors

 Data source Connector Package 

 Cloudant loopback-connector-cloudant 
 DashDB loopback-connector-dashdb 
 DB2 loopback-connector-db2 
 DB2 for zOS loopback-connector-db2z 
 Microsoft SQL Server loopback-connector-mssql
[Planning serverless apps](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-planplan)



* [Review the runtime support](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-planplan_runtime)
* [Architect your code](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-planplan_architect)
* [Determine your event source](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-planplan_source)
* [Organize deployments across environments](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-planplan_environments)



[Creating actions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions)



* [Creating actions from the CLI](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions_cli)



* [Combining app files and Docker images to create actions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions_combine_app)



* [Creating actions from the console](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions_create_ui)
* [Creating actions from binaries](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions_create_binaries)
* [Updating code or runtimes in actions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actionsactions_update)
refresh_token='<refresh-token>'
).get_result()

print(json.dumps(workspace_activity_plan_result, indent=2))

logging.info("Completed Generating Schematic Plan")





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 985
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "POST /v1/workspaces/us-south.workspace.Schematic-Sunil-Test-Workspace.5a4cbf11/plan HTTP/1.1" 202 49
{
  "activityid": "8c2ce5f031d23a60d26690d3cb398bb1"
}
INFO:root:Completed Generating Schematic Plan







 Next steps 

After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-applying-plan&interface=api) to deploy your IBM Cloud resources to build a Spectrum Scale cluster instance.