vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Usage monitoring 

You can monitor and track usage of your Discovery instance and use this data to help you understand and improve your applications. The [Events API](https://cloud.ibm.com/apidocs/discoverycreate-event) can be used to create log entries that are associated with specific natural language queries and actions. For example, you can record which documents in a results set were "clicked" by a user, and when that click occurred.

Logs and events are monitored only for natural language queries on private data collections. No logs are gathered on IBM Watson™ Discovery News.

Discovery can log the following information:



* Queries - Natural language queries run against collections in your environment
* Impressions (or Results) - The results returned for a particular query, usually documents or passages
* Events - Interactions a user performs on a result or set of results in Discovery (for example, a click on a document result)



Queries and Impressions are logged automatically; Events logging can be integrated into your application via the API.



 Viewing the logs 

You can search the query and event log to find query sessions that match the specified criteria. Searching the logs endpoint uses the standard Discovery query syntax for the parameters that are supported. The endpoint provides basic query functionality for viewing and searching through the recorded data.

The /api/v1/logs endpoint supports the following Discovery query parameters.



* query
* filter
* sort
* count
* offset
* version



For additional details on the function and syntax for the parameters see [Query parameters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameters).

Example of searching logs for a natural language query that contains the term “train”:

{url}/v1/logs?version=2019-04-30&query=train

Replace {url} with your URL.

The response of a logs query includes results that appear similar to Discovery document results. Each result is either a query or event, specified in the document type field.

Example query log:

{
"customer_id": "",
ibmcloud is load-balancers

For each load balancer in the subnet you want to delete, run the following command, where $lbid is the ID of the load balancer.

ibmcloud is load-balancer-delete $lbid

The status of the Load Balancer changes to deleting immediately, but it is still shown in a list query result. The deletion of a load balancer can take up to 30 minutes. You can request other subnet resources to be deleted in parallel while you wait for the load balancer to be deleted. However, the subnet can't be deleted until the load balancer and all other resources in the subnet no longer appear in the list query results.





 Delete all network interfaces in the subnet, if any 

An instance can have multiple network interfaces, and those network interfaces can belong to multiple subnets of the VPC. Before a subnet can be deleted, any network interface in the subnet must first be deleted.

The only way to delete a network interface in a subnet is to delete the instance.

To list all network interfaces of an instance, run the following command:

ibmcloud is instance-network-interfaces $vsi

To list all instances in your account, run the following command:

ibmcloud is instances

If you are deleting all instances in the VPC, you can run the following command for each instance in the VPC, where $vsi is the ID of the instance you want to delete. When the instance is deleted, all network interfaces in other subnets, if any, are deleted automatically.

Instances must be stopped before you can delete them. To stop an instance, run the ibmcloud is instance-stop command.

ibmcloud is instance-delete $vsi

The status of the instance changes to deleting immediately, but it still appears in a list query result. The deletion of an instance can take up to 30 minutes.

You can request other subnet resources to be deleted in parallel while you wait for the instance to be deleted. However, the subnet cannot be deleted until the instance and all other resources in the subnet no longer appear in the list query results.
location cos://us-geo/sql/LAX_Cargo.csv
options(HEADER=false)







 Drop Table 



 dropTable 

Drop a table definition from the catalog. If the table does not exist, you receive an error, unless the IF EXISTS option is specified.

This command does not delete any data in Object Storage. It removes only the table definition from the catalog.

-- drop a definition for the table customer
DROP TABLE customers







 Create view 



 createView 







 Drop view 



 dropView 

Drop a view definition from the catalog. If the view does not exist, you receive an error, unless the IF EXISTS option is specified.

Note: This command does not delete any data in Object Storage. It removes only the view definition from the catalog.

-- drop a view definition for the view customer_statistics
DROP VIEW customer_statistics







 Alter table partitions 



 alterTablePartitions 

Use alter table to modify the definition of the partitions or to automatically discover the available partitions.

Use the RECOVER PARTITIONS option to automatically replace the table partition metadata with the structure that is detected from Object Storage data by using the location prefix that is specified for the table.

-- replace the table partitions by scanning the location of the table
ALTER TABLE customers_partitioned RECOVER PARTITIONS





 partitionSpecification 

To add or remove partitions individually, use the ADD PARTITION or DROP PARTITION options.

With the ADD PARTITION option, you can specify an explicit location for the new partition. This way, you can construct a table from object locations that do not share a common Object Storage prefix, or are even located in separate buckets. If the partition location is not specified, it is inferred from the location of the table and the value(s) of the partitioning column(s). ADD PARTITION does not validate the specified or inferred location.

-- alter the table partitions by adding a partition
ALTER TABLE customers_partitioned ADD IF NOT EXISTS PARTITION ( COUNTRY = 'Spain') LOCATION cos://us-geo/sql/customers_partitioned.csv/COUNTRY=Spain
-- alter the table partitions by dropping a partition
* Be the ordering service admin. In which case you can add the peer organization you created to the consortium directly.
* Send MSP information to the ordering service admin, who will add your organization to their consortium.



In the next step, we will show how to add your peer organization to a consortium hosted by an ordering service in another IBM Blockchain Platform service instance. The tutorial assumes that you have created the ordering service and can follow every step yourself. If the ordering service was created in another console, ensure that the ordering service admin follows the steps marked with the blue "tip" box at the top.

If you followed the Build a network tutorial or if your console already includes the ordering service, you can skip ahead to [Add the peer's organization to the ordering service](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-join-networkibp-console-join-network-add-org2-local). You can then continue with [Step two: Add the peer's organization to an existing channel](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-join-networkibp-console-join-network-add-channel).





 Export your organization information 

If the ordering service admin does not already have your MSP, export it and send it to them now.





 Import the organization definition 

This step needs to be completed by an ordering service admin.

Once the MSP representing Org2 has been received, an administrator of the ordering service must import the JSON file by navigated to the Organizations tab, clicking the Import MSP definition button, and selecting the JSON file that represents the Org2 MSP peer organization MSP definition.





 Add Org2 MSP to the ordering service consortium 

This step needs to be completed by an ordering service admin.

After the MSP representing Org2 has been imported, use these steps to add the peer organization to the consortium hosted by the ordering service. Only ordering service admins, those who are listed under Ordering service administrators in the Details tab of the ordering service, can add peer organizations to the consortium.
Commit a smart contract definition After the proposal has reached the required number of approvals, the status of the proposal on the Channels tab changes to Ready to commit. Any channel member can click the tile and then click Commit to commit the definition to the channel. At that point, the associated smart contract can begin accepting transactions on the peers. 
 Update a smart contract definition that has been committed to the channel All updates to smart contract definitions have to be approved by channel members, according to the channel lifecycle endorsement policy, before they can be committed to the channel. Changes that require approval include:<br><br><br><br> * Updates to the agreed to business logic of the smart contract that require a new package.<br> * Updates to the smart contract endorsement policy.<br> * Updates to a private data collection.<br><br><br><br> <br> <br>Tip: If no changes are made to the agreed to business logic, then updates to the smart contract package do not require approval of channel members. See the next task, Update a smart contract package after the definition is committed to the channel. <br> <br>To update any of these elements, you have to create a new proposal in order for the update to be approved. You can additionally update the smart contract package but it is not required, for example if you only need to change the endorsement policy. <br> <br>Important: If the proposal includes a smart contract package that updates the agreed to business logic, all organizations that are part of the smart contract endorsement policy should install that package, or their own version of it, on their peers. Otherwise, after it is committed, the peers that are running the previous version are no longer able to endorse transactions. This could cause transactions not being committed to the ledger because there are not enough endorsements to satisfy the endorsement criteria. <br> <br>Because this is an update to an existing smart contract definition, you need to use the same smart contract definition name but it is recommended to increment the version by a second digit. For example, change the version from 1.1 to 1.2. This second digit version update is a best practice, which signals channel members that this proposal is an update to an existing smart contract definition and requires approval.
The smart contract definition includes the definition name, version, endorsement policy, and optionally, private data collections. As a proposer of the definition, Org1 attaches the smart contract package, which contains the agreed to business logic that updates the ledger. The package can be shared with the channel members out of band, who can review the source code. However, it's important to understand that channel members approve the smart contract definition, not the package. Because Org1 is proposing the smart contract definition, their approval is automatically submitted with the proposal. The other three organizations can approve or abstain. When channel members approve a smart contract definition, they are not required to install the package on their peers. They have three options:



* Simply approve the definition.
* Approve the definition and install the package on one or more of their peers.
* Approve the definition and upload their own package, as long as it does not change any agreed to business logic.



In the diagram, Org2 and Org4 approve the smart contract definition. Org3 abstains, perhaps because the owner of the Org3 admin identity is on vacation. After the approvals that are specified in the channel lifecycle endorsement policy are satisfied, any channel member that has approved the definition can commit it to the channel (only one channel member needs to perform the action). In this case, Org4 submits the commit transaction. It's important to note that Org3 can still come back later and approve the definition and install the package on the Org3 peers.

For more information about advanced channel options, which includes setting the lifecycle policy for the channel and the default endorsement policy that will be set for a smart contract if none is specified, check out the [Advanced options](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-governibp-console-govern-update-channel-available-parameters-advanced) section of our topic on channels.



 Before you begin 



 Ensure peer is running Fabric v2.x image 

As a reminder, to leverage the smart contract lifecycle process, the peers on the channel must be running a Fabric v2.x image.
Excluding ingestion data by using exclusion rules 

In an IBM Cloud Activity Tracker instance, you can configure exclusion rules through the UI to stop events from counting against your data usage quota and from being stored for search.



  Prereqs 

You must have manager access to define exclusion rules.





  Configuring an exclusion rule through the UI 

Complete the following steps to define an exclusion rule:

Verify that each exclusion rule that you add behaves as expected. Improper configured exclusion rules can result in storing data not intended for storage.



1.  [Launch the IBM Cloud Activity Tracker web UI](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-launch).
2.  Select the Settings icon ![Configuration icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/admin.png). Then select Usage > Exclusion Rules.
3.  Select Add Rule. The Create Rule section opens.
4.  Enter a name for the rule in the section What is this rule for?.
5.  Enter the exclusion criteria. You can select 1 or more hosts, 1 or more apps, enter a query, or a combination of hosts, apps and query.

For example, to exclude all the lines from a specific host, select that host and leave the apps and query fields blank. A host represents a service or resource.

You can enter a query to define the exclusion rule, or to refine the exclusion rule when you specify a host, an app, or both.
6.  Select Preserve these lines for live-tail and alerting to show through the live tail the log lines that are excluded. Notice that you can still use these log lines to set up an alert.
7.  Click Save.
8.  After you configure an exclusion rule, verify that the exclusion rule behaves as you expect.

Check the query in a custom view by entering the search criteria in the search bar of the Everything view, and validating that the data that is displayed is the data that you want excluded.
For example, to use the Secrets Manager tool integration, select Secrets Manager: ibm-secrets-manager-1. You can use other providers to manage your toolchain secrets, such as [HashiCorp Vault](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-hashicorpvault) and [Key Protect](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-keyprotect).
3. Select a secret group and a secret name to apply the stored secret to the field that is associated with it. Optional: select a secret key. Click OK. The name of the secret that you select appears in capsule form. You cannot edit the secret name inline, but you can click ![remove icon](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/secret-pill-delete-16.png) to delete the name. You can also replace the existing secret name by selecting the secret name again. If you manually type or paste a secret name into the Secrets field, it is displayed in a different format.









 Adding a Secrets Manager tool integration to your toolchain template 

You can add a Secrets Manager tool integration to your toolchain template by adding a service definition to the toolchain.yaml file in your template repo. This file is the design blueprint for your toolchain and includes all of the tool integrations that are available when you create a toolchain instance based on that template. To customize an existing toolchain template to include a Secrets Manager tool integration, insert a YAML definition.

sm-compliance-secrets:
service_id: secretsmanager
parameters:
name: sm-compliance-secrets
instance-id-type: instance-name
region: us-south
resource-group: default
instance-name: ffs-secrets







 Identifying secrets by CRN
To learn how to configure archiving for your logging instance, see [Archiving logs](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archiving).





 Step 8. Querying archived logs with SQL Query service 

Data Engine provides a serverless, no-ETL solution to easily query data stored in COS. [Learn more](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).

You can use this service to analyze data from logging archived files in COS.

Once you have the Data Engine service running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using Cloud Functions.

When you query logs, consider the following information:



* You must provision an instance of the Data Engine service in Frankfurt.
* You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using an EU-Supported bucket, create one. You can specify your custom bucket to store results in.
The endorsement policy specifies the set of organizations, the peers, on a channel that can execute the smart contract and independently validate the transaction output. For example, an endorsement policy can specify that a transaction is added to the ledger only if a majority of the members on the channel endorse the transaction. The organization that proposes the definition can select from among the members who have installed the smart contract to become validators, and sets the endorsement policy for all channel members. You can update your endorsement policy by following the steps for [updating your smart contract](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-smart-contracts-v2ibp-console-smart-contracts-v2-versioning).

When you follow the steps to [propose a smart contract definition](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-smart-contracts-v2ibp-console-smart-contracts-v2-install-propose), you can use the side panel to specify a contract's endorsement policy after selecting the smart contract package. You have the option to select the default endorsement policy inherited from the channel configuration or to select specific organizations that have to endorse the transactions as well as how many endorsements are required. For information about setting a default endorsement policy for a channel, check out the [Advanced options](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-governibp-console-govern-update-channel-available-parameters-advanced) section of our topic on channels.

Click the Advanced tab if you want to specify a custom policy in JSON format. Use this method when you need to specify more complicated endorsement policies, such as requiring that a certain member of the channel must validate a transaction, along with a majority of other members. You can more information about the required format in the [endorsement policy syntax](https://hyperledger-fabric.readthedocs.io/en/release-2.2/endorsement-policies.htmlendorsement-policy-syntax).

Endorsement policies are not updated automatically when new organizations join the channel and install a smart contract.
A connection error indicates that no HTTPS port is open on the service. Also, ensure that TLS connections can be terminated by your app. You can verify that your app terminates TLS properly by running curl -v https://<domain>. A certificate error indicates that your app is not properly terminating TLS connections.







 Persistent VPC load balancers 

By default, VPC load balancers are deleted when the cluster they are associated with is deleted. However, when you create a LoadBalancer service definition, you can make your load balancer persistent so that it remains available even after your cluster is deleted. A persistent VPC load balancer can be applied to a different cluster after its previous cluster is deleted.

The service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-lb-name annotation is supported only on cluster versions 1.24 and later.

VPC load balancer names are formatted as kube-<cluster_ID>-<kubernetes_lb_service_UID> by default. When a cluster is deleted, this name format specifies the associated load balancers that are then also deleted. To make sure that your load balancer is not deleted when you delete a cluster, include the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-lb-name annotation in your LoadBalancer service definition to give your load balancer a unique name. The load balancer name must be unique within your VPC, and can include only lowercase alphanumeric characters and dashes (-). The annotation can be applied to all VPC load balancer types.

You are responsible for deleting persistent VPC load balancers when they are no longer needed. To delete a persistent VPC load balancer, delete the Kubernetes LoadBalancer service definition that the VPC load balancer is associated with.





 Moving a VPC load balancer from one cluster to another 

[Persistent VPC load balancers](https://cloud.ibm.com/docs/containers?topic=containers-vpc-lbaasvpc_lb_persist) can be detached from one VPC cluster and then attached to another.
Command options 

--hostname HOST-NAME
: The hostname that will be set within the Hyper Protect Virtual Servers container by using this parameter value.

--rd REGISTRATION-DEFINITION
: The encrypted and signed registration definition that is used for [Bring your own server image (BYOI)](https://cloud.ibm.com/docs/hp-virtual-servers?topic=hp-virtual-servers-byoi). --rd or --rd-path is optional when you use a self-provided image.

--rd-path REGISTRATION-DEFINITION-PATH
: File path to the file that contains the encrypted and signed registration definition that is used for BYOI. --rd or --rd-path is optional when you use a self-provided image.

-i IMAGE-TAG
: The image tag for the BYOI server image. Required if you're using your own image.

-e ENV-CONFIG
: Specify environment variables when you use a self-provided image. They need to be specified in your registration definition first. You can set one or more environment variables as key value pairs by using the -e flag, for example, -ibmcloud hpvs instance-update CRN -i latest -e k1=v1 -e k2='v2 v3'. Environment variable names can have a maximum length of 64 characters and can be numbers, chars, and underscore. Environment variable values can have a maximum length of 4096.

--force
: Forces the update of the Hyper Protect Virtual Servers instance without prompting for confirmation.

To check the status of the update, run ibmcloud hpvs instance CRN and check the value of last operation in the output.







 hpvs instance-delete 

This command deletes a Hyper Protect Virtual Servers.

ibmcloud hpvs instance-delete (NAME | CRN)



 Command options 

NAME
: The name of your new instance. An error message is displayed if you specify a NAME that is not unique, for example,

VS name 'ABC' is ambiguous: more than one VS exist for the provided name: [crn1 crn2].