vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Using cloudyr for data science 

When you use the [R programming language](https://www.r-project.org/about.html) for your projects, get the most out of the features for supporting data science from IBM Cloud® Object Storage by using [cloudyr](https://cloudyr.github.io).

This tutorial shows you how to integrate data from the IBM Cloud® Platform within your R project. Your project will use IBM Cloud Object Storage for storage with S3-compatible connectivity in your project.



 Before you begin 

We need to make sure that we have the prerequisites before continuing:

- IBM Cloud Platform account
- An instance of IBM Cloud Object Storage
- R installed and configured
- S3-compatible authentication configuration



 Create HMAC credentials 

Before we begin, we might need to create a set of [HMAC credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main) as part of a [Service Credential](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-service-credentials) by using the configuration parameter {"HMAC":true} when we create credentials. For example, use the IBM Cloud Object Storage CLI as shown here.

ibmcloud resource service-key-create <key-name-without-spaces> Writer --instance-name "<instance name--use quotes if your instance name has spaces>" --parameters '{"HMAC":true}'

To store the results of the generated key, append the text, > cos_credentials to the end of the command in the example. For the purposes of this tutorial you need to find the cos_hmac_keys heading with child keys, access_key_id, and secret_access_key.

cos_hmac_keys:
access_key_id: 7xxxxxxxxxxxxxxa6440da12685eee02
For example, they might create spurious data, or delete valuable information. Protect these credentials carefully.

For more information about the fields included in the service credentials, see the [IAM guide](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudant).

In the next example, you create credentials for a service instance within IBM Cloud by running the following command.

ibmcloud resource service-key-create NAME ROLE_NAME --instance-name SERVICE_INSTANCE_NAME

The fields in the command are described in the table that follows.



Table 2. Create credential fields

 Field Description 

 NAME Arbitrary name that you give the service credentials. 
 ROLE_NAME This field currently allows the Manager role only. 
 SERVICE_INSTANCE_NAME The name that you give to your IBM Cloudant instance. 



Now, you create credentials for the cs20170517a instance you created in the previous step.



1. Create credentials for the cs20170517a instance of an IBM Cloudant service (where the name for the credentials is creds_for_cs20170517a) by running the following command.

ibmcloud resource service-key-create creds_for_cs20170517a Manager --instance-name cs20170517a
2. After you receive the request to create credentials for the service instance, review the response from IBM Cloud that contains a message similar to the one in the following example.

Creating service key in resource group default of account John Does's Account as john.doe@email.com...
OK
Service key crn:v1:bluemix:public:cloudantnosqldb:us-south:a/b42223455bb7e2abb0841ca25d28ee4c:ee78351d-82bf-4e80-bc22-825c937fafa3:resource-key:621ffde2-ea10-4318-b297-d6d849cec48a was created.

Name: creds_for_cs20170517a
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.





 Querying a view by using a list of keys 

You can also run a query by providing a list of keys to use.

Requesting information from a database in this way uses the specified $VIEW_NAME from the specified $DDOC design document. Like the keys parameter for the [GET](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsquerying-a-view) method, you can use the POST method to specify the keys to use to retrieve the view results. In all other aspects, the POST method is the same as the [GET](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsquerying-a-view) API request. In particular, you can use any of its query parameters in either the query string or the JSON body.

See the example HTTP request that returns all users, where the key for the view matches either amelie.smith@aol.com or bob.smith@aol.com:

POST $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME HTTP/1.1
Content-Type: application/json

{
"keys": [
"amelie.smith@aol.com",
"bob.smith@aol.com"
]
}

See the example of a global query that returns all users (where the key for the view matches is either amelie.smith@aol.com or bob.smith@aol.com):



* Curl
* Java
* Node
* Python
* Go
,"_deleted": true}]}'

You can also fetch many documents in a single request by issuing a POST to _all_docs (a relatively new endpoint that is called _bulk_get also exists, but this endpoint is probably not what you want. It’s there for a specific internal purpose).

To fetch a fixed set of docs by using _all_docs, POST with a keys body, run the following command:

curl -XPOST 'https://ACCT.cloudant.com/DB/_all_docs' -H "Content-Type: application/json" -d '{"keys":["ab234....","87addef...","76ccad..."]}'

IBM Cloudant (at the time of writing) imposes a max request size of 11 MB. _bulk_docs requests that exceed this size are rejected with a 413: Payload Too Large error.

For more information, see the following websites:



* IBM Cloudant [bulk operations docs](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documentsbulk-operations)
* IBM Cloudant [request and doc size limits](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-in-practicereplication-is-not-magic)







 Don’t mess with Q, R, and N unless you really know what you are doing 

Do not change Q, R, and N unless you really know what you're doing. IBM Cloudant’s quorum and sharding parameters, after you discover them, seem like tempting options to change the behavior of the database.

Stronger consistency, surely I can set the write quorum to the replica count?

No! Recall that no way exists to close the inconsistency window in a cluster.

Don’t go there. The behavior can be much harder to understand especially during network partitions. If you’re using Cloudant-the-service, the default values are fine for most users.
response = service.post_index(
db='readings',
name='timestamped-readings',
index=index,
type='json',
partitioned=True
).get_result()

print(response)

var indexField cloudantv1.IndexField
indexField.SetProperty("ts", core.StringPtr("asc"))

postIndexOptions := service.NewPostIndexOptions(
"readings",
&cloudantv1.IndexDefinition{
Fields: []cloudantv1.IndexField{
indexField,
},
},
)
postIndexOptions.SetName("timestamped-readings")
postIndexOptions.SetType("json")
postIndexOptions.SetPartitioned(true)

indexResult, response, err := service.PostIndex(postIndexOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(indexResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
"github.com/IBM/go-sdk-core/v5/core"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.





 Uploading partitioned index by device ID and timestamp 

Upload the index by device ID and timestamp to the database by using this command:



* Curl
* Java
* Node
* Python
* Go
"name": "my-index",
"ddoc": "my-index-design-doc",
"index": {
"default_field": {
"enabled": true,
"analyzer": "german"
},
"selector": {},
"fields": [
{"name": "married", "type": "boolean"},
{"name": "lastname", "type": "string"},
{"name": "year-of-birth", "type": "number"}
]
},
"partitioned": true
}
Show more



 The index field 

The index field contains settings specific to text indexes.

To index all fields in all documents automatically, use the simple syntax:

"index": {}

The indexing process traverses all of the fields in all the documents in the database.

In the [example movies' demo database](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-example-movies-demo-database), you can see an example of a text index that contains all fields and all documents in a database.

Take care when you index all fields in all documents for large data sets as it might be a resource-consuming activity.

See the following example of a JSON document that requests creation of an index of all fields in all documents:

{
"type": "text",
"index": { }
}





 The default_field field 

The default_field value specifies how the $text operator can be used with the index.

The default_field includes two keys:



Table 5. Default_field field keys

 Key Description 

 enabled Enable or disable the default_field index. The default value is true. 



The analyzer key in the default_field specifies how the index analyzes text. Later, the index can be queried by using the $text operator.
IBM Cloud Object Storage 

IBM Cloud Object Storage is a highly scalable cloud storage service, which is designed for high durability, resiliency, and security. Store, manage, and access your data through the self-service portal and RESTful APIs. The files that are uploaded into IBM Cloud Object Storage are called objects. The uploaded objects are organized into buckets that serve as containers for objects.

A IBM Cloud Object Storage represents a service destination, where an incoming notification can be stored and consumed programmatically to actions.



 Configuring a IBM Cloud Object Storage destination in the UI 

Before you configure IBM Cloud Object Storage as a destination, make sure that you have an IBM Cloud Object Storage instance [created and configured](https://cloud.ibm.com/objectstorage/create) in the same account as your Event Notifications instance.

If you are using Event Notifications CLI or API to configure IBM Cloud Object Storage service instance as a destination, ensure that you have enabled authorization to grant access between services before integrating with IBM Cloud Object Storage. For more information, see [Using authorizations to grant access between services](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-destinations-cloud-object-storageen-using-s2s-auth1).

To configure a IBM Cloud Object Storage destination, do the following steps:



1. From your Event Notifications instance dashboard, click Destinations.
2. Click Add + to add new destination.
3. In the Add a destination side panel, provide the following details.



* Name - Enter a name for your destination.
* Description - Optionally, enter a description for your destination.
* Type - Under Destination, for the Type, select IBM Cloud Object Storage from the list as your destination type.
* Instance name - Select the IBM Cloud Object Storage instance name from the list, if you already have an IBM Cloud Object Storage instance. Otherwise, click the Create new instance link, to create an IBM Cloud Object Storage instance.
[Unsaved changes are discarded](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/object-discard-changes.jpg)

As noted previously, you will have to authenticate in order to add tags to your data. If you have questions about bearer tokens, see [this example](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-curlcurl-token). Again, note the query string for working with tags: ?tagging.

curl -X "PUT" 'https://<endpoint>/<bucketname>/<objectname>?tagging' -H 'Authorization: bearer <token>' -H "content-type: text/plain" --data "<Tagging><TagSet><Tag><Key>your key</Key><Value>your text</Value></Tag></TagSet></Tagging>"

The example describes as shown a tag with a key of 'source' and a value of 'text' in the XML sent as data in the body of the request. The [schema](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-object-operationsobject-operations-add-tags) of the XML has to validate upon execution. If you want to add multiple tags, duplicate the 'Tag' node and modify the content of each key and value element to your specifications. There is only one 'TagSet' element for each object, and the 'PUT' command will replace any existing metadata with the values you specified.

curl -X "PUT" "https://s3.test.cloud-object-storage.sample.appdomain.cloud/taggingtest/example-file.csv?tagging" -H "Authorization: bearer ...iOiIyMDIwMTIwNzE0NDkiLCJh..."
* --json: Format output in JSON.



--------------------







 ibmcloud pi volume-clone 



 Get the status of a clone request for the specified clone task ID 

ibmcloud pi volume-clone CLONE_TASK_ID [--json]



* CLONE_TASK_ID: The unique identifier of a clone task.



Options



* --json: Format output in JSON.



--------------------







 ibmcloud pi volume-create-clone 



 Create a volume clone for specific volumes 

ibmcloud pi volume-create-clone CLONE_NAME --volumes "VOLUME1 ..VOLUMEn" [--json]



* CLONE_NAME: The name of a clone.



Options



* --volumes value Space: separated list of the volume(s) to be cloned.
* --json: Format output in JSON.



--------------------







 ibmcloud pi connections 



 List all cloud Connections 

ibmcloud pi connections [--long] [--json]

Options



* --long: Retrieve all cloud connections in detail
* --json: Format output in JSON



--------------------







 ibmcloud pi connection-attach-network 



 Attach a network to the cloud connection 

ibmcloud pi connection-attach-network CONNECTION_ID --network NETWORK_ID[--json]



* CONNECTION_ID: The unique identifier of the cloud connection



Options



* --network: The unique identifier (network ID) of the network



--------------------







 ibmcloud pi connection-create 



 Create a cloud connection
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

See the following example response after a text index is created successfully:

{
"id": "_design/bf936ad6eb87f03c2b42ae153377462844575e40",
"name": "bf936ad6eb87f03c2b42ae153377462844575e40",
"result": "created"
}

The most obvious difference in the results you get when you use full text indexes is the inclusion of a large bookmark field. The reason is that text indexes are different from view-based indexes. For more flexibility, when you work with the results that are obtained from a full text query, you can supply the bookmark value as part of the request body. Use the bookmark to specify which page of results you require.

The actual bookmark value is long, so the examples here have values that are truncated for reasons of clarity.

See the following example that uses HTTP to search for a specific document within the database:

POST /my-movies/_find HTTP/1.1
Host: $SERVICE_URL
Content-Type: application/json
{
"selector": {
"Person_name":"Zoe Saldana"
}
}

See the following example that uses the command line to search for a specific document within the database:



* Curl
* Java
* Node
* Python
* Go



curl -X POST -H "Content-Type: application/json" 	"$SERVICE_URL/my-movies/_find" 	-d '{"selector": {"Person_name":"Zoe Saldana"}}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.FindResult;
import com.ibm.cloud.cloudant.v1.model.PostFindOptions;

import java.util.Collections;
"url": "https://dbc68dd8-f69f-4083-97dd-bf0a3e1a467a-bluemix.cloudant.com/target",
"auth": {
"iam": {
"api_key": "UElc7...QIaL01Bjn"
}
}
},
"create_target": true,
"continuous": true
}
Show more

Now, write a replication document called source_dest to the _replicator database on the Source instance.

curl -k -X PUT -H"Content-type: application/json" -H'Authorization: '"$TOK"'' 'https://d43caf1b-e2c8-4d3e-9b85-1d04839fa68f-bluemix.cloudant.com/_replicator/source_dest' -d@data.json

See the results in the following example:

{"ok":true,"id":"source_dest","rev":"1-89b01e42968acd5944ed657b87c49f0c"}







 Removing IBM Cloudant legacy credentials from an instance 

IBM Cloud IAM is the recommended authentication model. For security purposes, you can request to remove the IBM Cloudant legacy credentials so that only IAM authentication can be used for the instance. The correct process to remove legacy credentials is shown in the following steps:



1. Ensure that the IBM Cloudant instance has IAM authentication that is enabled. If the instance is deployed in a Cloud Foundry org and space, migrate it to a Resource Group by using the [Resource Groups FAQ](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-faq-ibm-cloud-resource-groups).
2. Update your application to use IAM authentication instead of IBM Cloudant legacy authentication.
3.
The previous Go example requires the following import block: import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
"github.com/IBM/go-sdk-core/core"
All Go examples require the service object to be initialized. For more information, see the API documentation's Authentication section]] ! ! ! ! for examples.This option works, but you end up fetching n+1 documents when only n are required.<-- </section "id="section-option-1-fetch-one-doc-too-many" "> --><-- <section "id="section-option-2-the-u0000-trick" "> --> Option 2 - The \u0000 trick If you're determined to fetch only n documents each time, then you need to calculate a value of startkey, which means the next ID after the last _id in the result set. For example, if the last document in the first page of results is "example", what must the startkey of the next call to _all_docs be? It can't be "example", otherwise you get the same document ID again. It turns out that you can append u0000 to the end of a key string to indicate the "next key" (u0000 is a Unicode null character, which can be placed in a URL as-is or with the percent code %00). ).See the following example of a first request:<-- <ul> --> * Curl * Java * Node * Python * Go<-- </ul> --> curl -H "Authorization: Bearer $API_BEARER_TOKEN" "$SERVICE_URL/orders/_all_docs?limit=5"
import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.AllDocsResult;
import com.ibm.cloud.cloudant.v1.model.DocsResultRow;
The following example uses the policy ID and the option to receive the response in JSON format.

cloudshell:$ ibmcloud is backup-policy r138-0521986d-963c-4c18-992d-d6a7a99d115f --output JSON
{
"created_at": "2023-02-21T22:42:10.000Z",
"crn": "crn:v1:bluemix:public:is:eu-de:a/a10d63fa66daffc9b9b5286ce1533080::backup-policy:r138-0521986d-963c-4c18-992d-d6a7a99d115f",
"href": "https://eu-de.iaas.cloud.ibm.com/v1/backup_policies/r138-0521986d-963c-4c18-992d-d6a7a99d115f",
"id": "r138-0521986d-963c-4c18-992d-d6a7a99d115f",
"last_job_completed_at": "2023-02-22T20:12:44.000Z",
"lifecycle_state": "stable",
"match_resource_types": [
"volume"
],
"match_user_tags": [
"dev:test"
],
"name": "backup-policy-v1",
"plans": [
{
"href": "https://eu-de.iaas.cloud.ibm.com/v1/backup_policies/r138-0521986d-963c-4c18-992d-d6a7a99d115f/plans/r138-2129a79a-5629-4069-bf79-7bb0af3b0bd3",
The search is described by using a ['selector' expression](https://cloud.ibm.com/apidocs/cloudantpostfind) that looks like the following example:

{
"selector": {
"descriptionField": "freezing",
"temperatureField": -5
}
}

You can tailor the results by adding more details within the selector expression. The fields parameter specifies the fields to include with the results. In our example, the results include the nameField, descriptionField, and temperatureField, as shown in the following example.

{
...
"fields": [
"nameField",
"descriptionField",
"temperatureField"
]
}



To run the query, follow these steps:



1. Copy the sample JSON into a data file named query2.json.

{
"selector": {
"descriptionField": "freezing",
"temperatureField": -5
},
"fields": [
"nameField",
"descriptionField",
"temperatureField"
]
}
2. Run the following command to query the database:

acurl "https://$ACCOUNT.cloudant.com/databasedemo/_find" -X POST -H "Content-Type: application/json" -d @query2.json
3. Review the query results:

{
"docs": [{
"nameField": "eight",
"descriptionField": "freezing",
"temperatureField": -5
}],
"bookmark": "g1AAAABweJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYorpBiYGJolWaSZmaWYpFqmGBiYmKclphhappqZmRglG5uD9HHA9BGlIwsAms8eJw",
"warning": "No matching index found, create an index to optimize query time."
}



Next, you find a document in the database by using multiple operators.





 Step 4: Running a query with operators 

In this example, the $gt (greater than) and $eq (equal) operators are used to search for documents that include a temperature that is greater than 20 degrees and a description that contains the value hot.
For Available authentication methods, select Use both legacy credentials and IAM. You can leave the default settings for the other fields. Click Create to create the service.
4. In the navigation, go to Connections, then click Create connection. Select your app, and click Connect.
5. Using the default values, click Connect & restage app to connect the database to your app. Click Restage when prompted.



IBM Cloud will restart your app and provide the database credentials to your app using the VCAP_SERVICES environment variable. This environment variable is available to the app only when it is running on IBM Cloud.

Environment variables enable you to separate deployment settings from your source code. For example, instead of specifying a database password in your source code, you can store it in an environment variable that you reference in your source code.





 Step 6: Use the database 

We're now going to update your local code to point to this database. We'll create an .env file that will store the credentials for the services the app will use. This file will get used ONLY when the app is running locally. When running in IBM Cloud, the credentials will be read from the VCAP_SERVICES environment variable.



1. Create a file called .env in the get-started-php directory with the following content:

CLOUDANT_HOST=
CLOUDANT_USERNAME=
CLOUDANT_PASSWORD=
2. Find your app in the IBM Cloud [resource list](https://cloud.ibm.com/resources). On the Service Details page for your app, click Connections in the sidebar. Click the IBM Cloudant menu icon (…) and select View credentials.
3. Copy and paste values of the CLOUDANT_HOST, CLOUDANT_USERNAME and CLOUDANT_PASSWORD fields into the .env file and save the changes. The result will be something like:

CLOUDANT_HOST=abc...yz.cloudant.com
CLOUDANT_USERNAME=abc...yz
CLOUDANT_PASSWORD=445d...d1a
4. Run your app locally.

php -S localhost:8000