vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* To search for resources that are not classic infrastructure and have been tagged with MyTag, enter the following command:

ibmcloud resource search 'tags:MyTag'
* To search for all classic infrastructure virtual servers whose fully qualified domain name is MyVM, enter the following command:

ibmcloud resource search “doc.fullyQualifiedDomainName:MyVM AND service_name:virtual-server”
* To search for all classic infrastructure resources that have been tagged with MyTag, enter the following command:

ibmcloud resource search 'tagReferences.tag.name:MyTag' -p classic-infrastructure
* To search for all classic infrastructure of type SoftLayer_Network_Vlan

ibmcloud resource search '_objectType:SoftLayer_Network_Vlan' -p classic-infrastructure









 Search by using the API 

To search for resources, call [The Global Search and Tagging - Search API](https://cloud.ibm.com/apidocs/searchsearch). The following example searches for all resources with tag "project:myproject" attached.

Use the SearchOptions.Builder to create a SearchOptions object that contains the parameter values for the search method.

Instantiate the SearchOptions struct and set the fields to provide parameter values for the Search method.



* Curl
* Java
* Python
* Go
* Node



curl -X POST -H "Authorization: {iam_token}" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"query": "tags:project\:myproject OR access_tags:project\:myproject", "fields": [""]}' "api.global-search-tagging.cloud.ibm.com/v3/resources/search"

SearchOptions searchOptions = new SearchOptions.Builder()
.query("GST-sdk-")
.fields(new java.util.ArrayList<String>(java.util.Arrays.asList("")))
.searchCursor(searchCursor)
.build();
"Parts": data_packs
}
)
print("Upload for {0} Complete!n".format(item_name))
except ClientError as be:
 abort the upload
cos_cli.abort_multipart_upload(
Bucket=bucket_name,
Key=item_name,
UploadId=upload_id
)
print("Multi-part upload aborted for {0}n".format(item_name))
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
print("Unable to complete multi-part upload: {0}".format(e))
Show more

SDK References



* Classes



* [S3.Client](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlclient)



* Methods



* [abort_multipart_upload](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Client.abort_multipart_upload)
* [complete_multipart_upload](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Client.complete_multipart_upload)
* [create_multipart_upload](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Client.create_multipart_upload)
* [upload_part](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Client.upload_part)











 Large Object Upload by using TransferManager 

The TransferManager provides another way to run large file transfers by automatically incorporating multi-part uploads whenever necessary setting configuration parameters.
s3cmd put bigfile.pdf s3://backuptest/bigfile.pdf --multipart-chunk-size-mb=5

Output:

upload: 'bigfile.pdf' -> 's3://backuptest/bigfile.pdf' [part 1 of 4, 5MB] [1 of 1]
5242880 of 5242880 100% in 2s 1731.92 kB/s done
upload: 'bigfile.pdf' -> 's3://backuptest/bigfile.pdf' [part 2 of 4, 5MB] [1 of 1]
5242880 of 5242880 100% in 2s 2001.14 kB/s done
upload: 'bigfile.pdf' -> 's3://backuptest/bigfile.pdf' [part 3 of 4, 5MB] [1 of 1]
5242880 of 5242880 100% in 2s 2000.28 kB/s done
upload: 'bigfile.pdf' -> 's3://backuptest/bigfile.pdf' [part 4 of 4, 4MB] [1 of 1]
4973645 of 4973645 100% in 2s 1823.51 kB/s done





 Using the Java SDK 

The Java SDK provides two ways to run large object uploads:



* [Multipart Uploads](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-javajava-examples-multipart-object)
* [TransferManager](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-javajava-examples-transfer-manager)







 Using the Python SDK 

The Python SDK provides two ways to run large object uploads:



* [Multipart Uploads](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-pythonpython-examples-multipart)
<Expiration>
<Days>60</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Show more

Code sample for use with NodeJS COS SDK

Using the IBM Cloud® Object Storage SDKs only requires calling the appropriate functions with the correct parameters and proper configuration.

var aws = require('ibm-cos-sdk');
var ep = new aws.Endpoint('s3.us-south.cloud-object-storage.appdomain.cloud');
var config = {
endpoint: ep,
apiKeyId: 'ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE',
ibmAuthEndpoint: 'https://iam.cloud.ibm.com/identity/token',
serviceInstanceId: 'crn:v1:bluemix:public:cloud-object-storage:global:a/<CREDENTIAL_ID_AS_GENERATED>:<SERVICE_ID_AS_GENERATED>::',
};
var s3 = new aws.S3(config);
var date = new Date('June 16, 2019 00:00:00');

var params = {
Bucket: 'STRING_VALUE', /* required /
LifecycleConfiguration: {
Rules: [ /* required /
{
Status: 'Enabled', /* required /
ID: 'OPTIONAL_STRING_VALUE',
Filter: {}, /* required /
Expiration:
{
Date: date
}
},
]
}
};

s3.putBucketLifecycleConfiguration(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});

Code sample for use with Python COS SDK
* [get](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Object.get)









 Delete an item from a bucket 

def delete_item(bucket_name, object_name):
try:
cos.delete_object(Bucket=bucket_name, Key=object_name)
print("Item: {0} deleted!n".format(object_name))
except ClientError as be:
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
print("Unable to delete object: {0}".format(e))

SDK References



* Classes



* [Object](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlobject)



* Methods



* [delete](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Object.delete)









 Delete multiple items from a bucket 

The delete request can contain a maximum of 1000 keys that you want to delete. While this is useful in reducing the per-request performance hit, be mindful when deleting many keys. Also, take into account the sizes of the objects to ensure suitable performance.

def delete_items(bucket_name):
try:
delete_request = {
"Objects": [
{ "Key": "deletetest/testfile1.txt" },
{ "Key": "deletetest/testfile2.txt" },
{ "Key": "deletetest/testfile3.txt" },
{ "Key": "deletetest/testfile4.txt" },
{ "Key": "deletetest/testfile5.txt" }
]
}

response = cos.delete_objects(
Bucket=bucket_name,
* [TransferConfig](https://ibm.github.io/ibm-cos-sdk-python/reference/customizations/s3.htmls3-transfers)



* Methods



* [upload_fileobj](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Object.upload_fileobj)









 Manually run a multi-part upload 

If wanted, the [S3.Client](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlclient) class can be used to perform a multi-part upload. This can be useful if more control over the upload process is necessary.

def multi_part_upload_manual(bucket_name, item_name, file_path):
try:
 create client object
cos_cli = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_SERVICE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)

print("Starting multi-part upload for {0} to bucket: {1}n".format(item_name, bucket_name))

 initiate the multi-part upload
mp = cos_cli.create_multipart_upload(
Bucket=bucket_name,
Key=item_name
)

upload_id = mp["UploadId"]

 min 20MB part size
part_size = 1024 * 1024 * 20
file_size = os.stat(file_path).st_size
part_count = int(math.ceil(file_size / float(part_size)))
data_packs = []
position = 0
part_num = 0

 begin uploading the parts
* <service-instance-id> - resource ID for your cloud Object Storage (available through [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-idt-cli) or [IBM Cloud Dashboard](https://cloud.ibm.com/resources))
* <location> - default location for your cloud Object Storage (must match the region that is used for <endpoint>)



SDK References



* [ServiceResource](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlservice-resource)







 Creating a new bucket 

A list of valid provisioning codes for LocationConstraint can be referenced in [the Storage Classes guide](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-classesclasses).

def create_bucket(bucket_name):
print("Creating new bucket: {0}".format(bucket_name))
try:
cos.Bucket(bucket_name).create(
CreateBucketConfiguration={
"LocationConstraint":COS_BUCKET_LOCATION
}
)
print("Bucket: {0} created!".format(bucket_name))
except ClientError as be:
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
print("Unable to create bucket: {0}".format(e))

SDK References



* Classes



* [Bucket](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlbucket)



* Methods



* [create](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Bucket.create)









 Creating a new text file 

def create_text_file(bucket_name, item_name, file_text):
VersioningConfiguration={
'Status': 'Enabled'
}
)
Show more

Listing the versions of an object using the same client:

resp = cosClient.list_object_versions(Prefix='some-prefix', Bucket=BUCKET)

Note that the Python APIs are very flexible, and there are many different ways to accomplish the same task.





 Node.js 

Enabling versioning using the [IBM COS SDK for Node.js](https://ibm.github.io/ibm-cos-sdk-js/AWS/S3.htmlputBucketVersioning-property):

const IBM = require('ibm-cos-sdk');

var config = {
endpoint: '<endpoint>',
apiKeyId: '<api-key>',
serviceInstanceId: '<resource-instance-id>',
};

var cos = new IBM.S3(config);

var params = {
Bucket: 'my-versioning-bucket', /* required /
VersioningConfiguration: { /* required /
Status: 'Enabled'
},
};

s3.putBucketVersioning(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
Show more
ibm_api_key_id=COS_API_KEY_ID,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT,
ibm_service_instance_id=COS_RESOURCE_INSTANCE_CRN,
ibm_auth_endpoint="https://iam.test.cloud.ibm.com/identity/token"
)
new_bucket_name = "create-example-python12345" bucket name should be unique gloablly, or else it will throw an error.
new_text_file_name = "cos_object.txt"
new_text_file_contents = "This is a test file from Python code sample!!!"

 Main Program 
def main():
create_bucket_with_objectlock(new_bucket_name) Create a new cos bucket with object lock enabled.
objectlock_configuration_on_bucket(new_bucket_name) Put objectlock configuration(i.e. default retention) on COS bucket and get the configuration.
upload_object(new_bucket_name,new_text_file_name,new_text_file_contents) Upload an object to cos bucket.
objectlock_retention(new_bucket_name,new_text_file_name) Put objectlock retention(i.e. retain until date) on the object and get the configured retention.
objectlock_legalhold(new_bucket_name,new_text_file_name) Put objectlock legalhold on the object and get the legalhold status.

if __name__ == "__main__":
main()

Show more





 Node.js 

Enabling versioning using the [IBM COS SDK for Node.js](https://ibm.github.io/ibm-cos-sdk-js/AWS/S3.htmlputBucketVersioning-property):

'use strict';

// Required libraries
const ibm = require('ibm-cos-sdk');
A package.json file must be in your Node.js app for the app to be recognized by the Node.js buildpack. The app.js file is the entry script for the app, and can be specified in the package.json file. The following example shows a simple package.json file:

{
"name": "MyUniqueNodejs01",
"version": "0.0.1",
"description": "A sample package.json file",
"dependencies": {
"express": ">=3.4.7 <4",
"jade": ">=1.1.4"
},
"scripts": {
"start": "node app.js"
},
"engines": {
"node": ">=0.10.0"
},
"repository": {}
}
Show more

For more information about the package.json file, see [Creating a package.json file](https://docs.npmjs.com/creating-a-package-json-file).
* To deploy PHP, Ruby, or Python apps to IBM Cloud, use the following command from the directory that contains your app source:

ibmcloud cf push <appname>





 Deploying an app in multiple spaces 

An app is specific to the space where it is deployed. You can't move or copy an app from one space to another in IBM Cloud. To deploy an app in multiple spaces, you must deploy your app in each space where you want to use the app by the following steps:



1. Switch to the space where you want to deploy your app by using the ibmcloud cf target command with the -s option:

ibmcloud cf target -s <space_name>
2. Go to your app directory and deploy your app by using the ibmcloud cf app push command, where <appname> must be unique within your domain.

ibmcloud cf app push <appname>









 App manifest
Date: date
}
},
]
}
};

s3.putBucketLifecycleConfiguration(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});

Code sample for use with Python COS SDK

Using the IBM Cloud® Object Storage SDKs only requires calling the appropriate functions with the correct parameters and proper configuration.

import sys
import ibm_boto3
from ibm_botocore.client import Config

api_key = "ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE"
service_instance_id = "85SAMPLE-eDOb-4NOT-bUSE-86nnnb31eaxx"
auth_endpoint = "https://iam.cloud.ibm.com/identity/token"
service_endpoint = "https://s3.us-south.cloud-object-storage.appdomain.cloud"

cos = ibm_boto3.client('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)

response = cos.put_bucket_lifecycle_configuration(
Bucket='string',
LifecycleConfiguration={
'Rules': [
{
'Status': 'Enabled',
'Filter': {},
'Expiration':
{
'Days': 123
},
},
]
}
)

print("Bucket lifecyle: {0}".format(response))
Show more

Code sample for use with Java COS SDK
The app and the associated eventing allow to automatically retrieve traffic data for GitHub repositories. Information about those repositories, including the tenant-specific access token, is stored in a SQL database (Db2® Warehouse on Cloud). That database is used by the Python app to manage users, repositories and to present the traffic statistics. Users can see the traffic statistics in searchable tables or visualized in a simple line chart (see image below). It is also possible to download the list of repositories and the traffic data as CSV files.

Zoom

![Line chart](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution24-github-traffic-analytics/LineChart.png)

Line chart





 Step 8: Security: Rotate service credentials 

If you use this solution in production, then you should rotate the service credentials on a regular basis. Many security policies have a requirement to change passwords and credentials every 90 days or with similar frequency.

You can recreate and thereby rotate the credentials for the services bound to the app by unbinding, then binding the services again. When using secrets instead of service bindings, you even have more options by first recreating service keys, then updating the secrets and as last step updating the app.





 Step 9: Remove resources 

To clean up the resources used for this tutorial, you can delete the related project and services.



1. Unbind the the provisioned services. First display the bindings then delete them by Service Bindings Names (FIRST and SECOND below are from the get output)

ibmcloud ce application get --name ghstats-app

ibmcloud ce application unbind --name ghstats-app --binding ghstats-app-ce-service-binding-FIRST

ibmcloud ce application unbind --name ghstats-app --binding ghstats-app-ce-service-binding-SECOND
2.
See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless).

To use the IBM Analytics Engine Java SDK, you need the following values:



* IAM_API_KEY: The API key generated when creating the service credentials. You can retrieve by viewing the service credentials on the [IBM Cloud dashboard](https://cloud.ibm.com/resources).
* instance_guid: The value in resource_instance_id generated when the service credentials are created. You can get the value by viewing the service credentials on the [IBM Cloud dashboard](https://cloud.ibm.com/resources).
* IAE_ENDPOINT_URL : The service endpoint URL including the https:// protocol. See [Service endpoints](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3?code=javaservice-endpoints).







 Initializing the configuration 

The Java SDK allows you to construct the service client in one of two ways by:



* By setting the client options programmatically

You can construct an instance of the IBM Analytics Engine service client by specifying various client options, like the authenticator and service endpoint URL, programmatically:

import com.ibm.cloud.iaesdk.ibm_analytics_engine_api.v3.IbmAnalyticsEngineApi;
import com.ibm.cloud.iaesdk.ibm_analytics_engine_api.v3.model.;
import com.ibm.cloud.sdk.core.http.Response;
import com.ibm.cloud.sdk.core.security.;
import java.util.HashMap;

private static IbmAnalyticsEngineApi ibmAnalyticsEngineApiService;

private static String IAM_API_KEY = "{apikey}";
private static String IAE_ENDPOINT_URL = "{url}";
private static String API_AUTH_URL = "{api auth url}";
[Vertical overflow menu icon](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/overflow-menu--vertical.svg) from the page header, and choose Export model.
11. Save the .sdumodel file to your system in a location where you can access it again shortly.
12. From the navigation panel, click Manage collections, and then open the Discovery docs part 2 collection.
13. Open the Identify fields tab.
14. Click User-trained models, and then click Submit.
15. Click Apply changes and reprocess.
16. From the SDU editor toolbar, click the overflow menu icon ![Vertical overflow menu icon](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/overflow-menu--vertical.svg), and choose Import model, and then click Select model.
17. Browse to find the .sdumodel file that you downloaded earlier, and then click Open.
18. Click Apply changes and reprocess to apply the same SDU model to the first collection.



Discovery reprocesses the data in its index to identify subtitles in the documents. While the data is being reprocessed, let's create our answer generator.





 Step 6: Create a NeuralSeek service instance 

You can use a search extension in Watson Assistant to connect your assistant directly to Discovery and return passages straight from the data source. However, we will add the NeuralSeek service between Watson Assistant and Discovery in this tutorial. NeuralSeek retrieves the passages from Discovery and then converts them into answers that sound more conversational.



1. From a new web browser tab, return to the IBM Cloud catalog.
pid @{HOME}/nginx/logs/nginx.pid;

 How to fix it 

The NOTICE messages are for information and might not indicate a problem. You can stop the logging of these messages by changing the logging level from stderr notice to stderr error in the nginx-defaults.conf file in your buildpack. For example:

daemon off;
error_log stderr error;
pid @{HOME}/nginx/logs/nginx.pid;

For more information about how to change the default logging configuration, see [error_log](http://nginx.org/en/docs/ngx_core_module.htmlerror_log).







 Python 



 Can't import a third-party Python library into IBM Cloud 

You might not be able to import a third-party Python library into IBM Cloud. To resolve the problem, add configuration files to the root directory of your python app.

 What’s happening 

When you try to import a third-party Python library, such as the web.py library, the ibmcloud cf push command fails.

 Why it’s happening 

Configuration information for the Python app is missing.

 How to fix it 

Add a requirements.txt file and a Procfile file to the root directory of your Python app. The following information assumes that you are importing the web.py library:



1. Add a requirements.txt file to the root directory of your Python app.

The requirements.txt file specifies the library packages that are required for your Python app and the version of the packages. The following example shows the content of the requirements.txt file, where web.py==0.37 indicates the version of the web.py library that will be downloaded is 0.37, and wsgiref==0.1.2 indicates the version of the web server gateway interface that is required by the web.py library is 0.1.2.

web.py==0.37
wsgiref==0.1.2
file: fs.createReadStream('./test-doc1.html'),
};

discovery.addDocument(addDocumentParams)
.then(response => {
const documentAccepted = response.result;
console.log(JSON.stringify(documentAccepted, null, 2));
})
.catch(err => {
console.log('error:', err);
});
Show more
* Python:

import json
from ibm_watson import DiscoveryV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator

authenticator = IAMAuthenticator('{apikey}')
discovery = DiscoveryV1(
version='2019-04-30',
authenticator=authenticator
)

discovery.set_service_url('{url}')

with open('./test-doc1.html') as fileinfo:
add_doc = discovery.add_document(
'{environment_id}',
'{collection_id}',
file=fileinfo).get_result()
print(json.dumps(add_doc, indent=2))
Show more







1. Repeat this process for each of the other 3 sample files.







 Step 5: Query your collection 

Finally, use the GET /v1/environments/{environment_id}/collections/{collection_id}/query method to search your collection of documents.

The following example returns all entities that are called IBM. Replace {apikey}, {environment_id} and {collection_id} with your information:

curl -u "apikey:{apikey}" "{url}/v1/environments/{environment_id}/collections/{collection_id}/query?version=2019-04-30&query=enriched_text.entities.text:IBM"





 Next steps 

You successfully queried documents in the environment and collection you created.
If you want to update multiple values at the same time, follow the steps for [Updating variables with Schematics API](https://cloud.ibm.com/docs/hpc-openshift?topic=hpc-openshift-update-variables).





 Example Python request 

 Create a Schematics workspace using Python API

import json, logging

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_schematics.schematics_v1 import SchematicsV1

logging.basicConfig()
logging.root.setLevel(logging.NOTSET)
logging.basicConfig(level=logging.NOTSET)

authenticator = IAMAuthenticator('')

schematics_service = SchematicsV1(authenticator = authenticator)

schematics_service.set_service_url('https://us.schematics.cloud.ibm.com')

template_source_data_request_model = {}
template_source_data_request_model['env_values'] = [{'Environment': 'openshift-hpcc'}]
template_source_data_request_model['folder'] = ''
template_source_data_request_model['type'] = 'terraform_v1.0'
template_source_data_request_model['variablestore'] = []

template_repo_request_model = {}
template_repo_request_model['url'] = 'https://github.com/IBM-Cloud/hpc-cluster-openshift'

logging.info("Started Creating Schematic Workspace")

workspace_response = schematics_service.create_workspace(
description="HPC Cluster schematic workspace using API",
name="Sample Schematic API workspace",
template_data=[template_source_data_request_model],
template_repo=template_repo_request_model,