vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Enter the following command to expand the custom-crawler-docs.zip file:

unzip custom-crawler-docs.zip -d custom-crawler-docs-primary

If necessary, copy the custom-crawler-docs.zip file to the development server.

If your local machine does not have the unzip utility, try using the gunzip command instead, or see the documentation of the operating system of your local machine for other alternatives to expand compressed files.

If you are using a version of Discovery that is earlier than 2.1.2 and you want to access the custom-crawler-docs.zip file, enter the following command: scp root@{instance_name}:/root/bob/sdk/custom-crawler-docs.zip {local_directory}.



For information about downloading the custom-crawler-docs.zip file to Discovery 2.2.0 and earlier, see [Downloading the custom-crawler-docs.zip file to Discovery 2.2.0 and earlier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-devdownload-ccs-zip-old).



 Downloading the custom-crawler-docs.zip file to Discovery 2.2.0 and earlier 

In Discovery version 2.2.0 and earlier, perform the following steps to download the custom-crawler-docs.zip file to your local machine. You need root access to an installed Discovery instance:



1. Obtain the entitlement key by navigating to your [container software library](https://myibm.ibm.com/products-services/containerlibrary).
2. Enter the following command to log in to the Docker registry where your Discovery images are available. Include your entitlement key in the following command:

docker login cp.icr.io -u cp -p {entitlement_key}
3. Enter the following command to pull the custom-crawler-sdk image:

docker pull cp.icr.io/cp/watson-discovery/custom-crawler-sdk:2.1.3
4.
Terraform takes all of the required actions to create the tool integration.

terraform apply







 Updating a tool integration by using the console 

If you deferred the configuration of a tool integration when you created a toolchain, a Configure button is shown on its card. If you configured a tool integration when you created a toolchain, you can update the configuration settings by using the console.



1. From the IBM Cloud console, click the menu icon ![hamburger icon](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/icon_hamburger.svg), and select DevOps.
2. On the Toolchains page, click the toolchain that contains the tool integration that you want to update to open its Overview page. Alternatively, on the App details page in your app, click the toolchain name.
3. If you need to configure a tool integration for the first time, on its card, click Configure.
4. When you are finished configuring the tool integration, click Save Integration.
5. If you need to update a tool integration's configuration, click the Actions icon ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/icons/action-menu-icon.svg)> Configure.
6. When you are finished updating the settings, click Save Integration.







 Updating a tool integration with the API 

If you configured a tool integration when you created a toolchain, you can update the configuration settings with the API.



1. [Obtain an IAM bearer token](https://cloud.ibm.com/apidocs/toolchainauthentication). Alternatively, if you are using an SDK, [obtain an IAM API key](https://cloud.ibm.com/iam/apikeys) and set the client options by using environment variables.
* [CMIS (Content Management Interoperability Services)](https://cloud.ibm.com/docs/discovery?topic=discovery-configuring-connector-and-seed-optionscmis-crawl-options)
* [SMB (Server Message Block), CIFS (Common Internet Filesystem), or Samba fileshares](https://cloud.ibm.com/docs/discovery?topic=discovery-configuring-connector-and-seed-optionssmb-cifs-samba-crawl-options)



3. Save and close the file.
4. Repeat for the -seed.conf file in the connectors/seeds directory that corresponds to the repository you are connecting to (for example filesystem-seed.conf is the seed (where to connect to) configuration file for the filesystem connector) in a text editor.
5. Proceed to [Configuring the Data Crawler](https://cloud.ibm.com/docs/discovery?topic=discovery-configuring-the-data-crawler).



To access the in-product manual for the connector and seed configuration files, with the most up-to-date information, type the following commands from the Crawler installation directory:



* For connector configuration options:

man crawler-options.conf
* For crawl seed configuration options:

man crawler-seed.conf





 Configuring filesystem crawl options 

The filesystem connector allows you to crawl files local to the Data Crawler installation.

Another option to upload large numbers of files into Discovery is [discovery-files](https://github.com/IBM/discovery-files) on GitHub.



 Configuring the filesystem connector 

Following are the basic configuration options that are required to use the filesystem connector. To set these values, open the file config/connectors/filesystem.conf, and modify the following values specific to your use cases:



* protocol - The name of the connector protocol used for the crawl. Use sdk-fs for this connector.
* collection - This attribute is used to unpack temporary files.
First, you need to export your connection profile:



1. Ensure that the network is running in the Fabric Environments pane.
2. Right-click on the gateway in the Fabric Gateways panel and select Export Connection Profile.



You can then use the Fabric SDKs and the connection profile to enroll your admin identity by using the username admin and the password adminpw. You can then use this identity to invoke your smart contract or register and enroll additional users.







 Step five: Test a deployed smart contract 

You can generate tests for smart contracts that are instantiated or committed on the channels that you connect to. The tests can be generated as either JavaScript or TypeScript for Node projects, Java for Java projects, or Go for Go projects. Generated tests can then be run or debugged.



1. Ensure that the smart contract is instantiated or committed on the channel.
2. In the Fabric Gateways pane, right-click the smart contract under the list of channels to generate tests for.
3. Select Generate Smart Contract Tests.
4. Select the language for the test file, either JavaScript, TypeScript, Java, or Go, depending on the smart contract language. The IBM Blockchain Platform extension installs the required dependencies and builds the test file.
5. Make sure you have either the [Node Test Runner extension](https://marketplace.visualstudio.com/items?itemName=oshri6688.javascript-test-runner), [Java Test Runner extension](https://marketplace.visualstudio.com/items?itemName=vscjava.vscode-java-test) or [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed.



After the test file is built, the tests can be run by clicking the Run Tests button in the file.





 Step six: Connect to your IBM Blockchain Platform network 

You can also use the extension to interact with your network on the IBM Blockchain Platform.



 Invoke a smart contract that has been instantiated or committed on your channels
bmsClient.initialize(getApplicationContext(), <region>);
AppIDAuthorizationManager appIdAuthMgr = new AppIDAuthorizationManager(AppID.getInstance())
bmsClient.setAuthorizationManager(appIdAuthMgr);
Request request = new Request("{your protected resource url}", Request.GET);
request.send(this, new ResponseListener() {
@Override
public void onSuccess (Response response) {
Log.d("My app", "onSuccess :: " + response.getResponseText());
}
@Override
public void onFailure (Response response, Throwable t, JSONObject extendedInfo) {
if (null != t) {
Log.d("My app", "onFailure :: " + t.getMessage());
} else if (null != extendedInfo) {
Log.d("My app", "onFailure :: " + extendedInfo.toString());
} else {
Log.d("My app", "onFailure :: " + response.getResponseText());
}
}
});
Show more







 Without an SDK 

With the library of your choice, set your Authorization request header to use the Bearer authentication scheme to transmit the access token.

Example request format:

GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer <accessToken> <optionalIdentityToken>







 Next steps 

With App ID installed in your application, you're almost ready to start authenticating users! Try doing one of the following activities next:



* Configure your [identity providers](https://cloud.ibm.com/docs/appid?topic=appid-social)
Properties 

You can create and manage properties, add them to collections and use them in your app by using SDKs. You can use them in your infrastructure by using [terraform plug-in](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/app_config_collection) or CLI. Using properties, you can manage the configuration properties of your infrastructure or applications centrally by using App Configuration.

By default, the properties window lists all properties in an environment in the current App Configuration service instance. Attributes for each property like Name, Value, Date of creation, Last evaluated, Segments that are targeted, and Type are displayed.

Property types:



* A Boolean type has two values and you can set the default value as 'true' or 'false'.
* A String type property supports data in the form of text, and JSON or YAML files.
* A Numeric property supports integer values.
* A Secret reference property supports Secrets Manager integration.

For selecting Secret reference property, you need have a Secrets Manager instance with Viewer access to the resource group where Secret Manager instance is created or exists. Also, you need to have Reader access to the Secrets Manager instance.



An example use case of properties can be to decide the number of instances of your application in a specific region. Create a property in App Configuration, with type as numeric, and assign segments based on region value.

Zoom

![List of properties](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-properties-default.png)

Figure 1. List of properties



 Create a property 

To create a property, complete these steps:



1. In the App Configuration console, click Properties.
2. Select the Current Environment to display the list of properties in that environment.
3. Click Create. The side panel opens with fields for creating a new property.

Zoom

!
Configuring the filesystem connector 

Following are the basic configuration options that are required to use the filesystem connector. To set these values, open the file config/connectors/filesystem.conf, and modify the following values specific to your use cases:



* protocol - The name of the connector protocol used for the crawl. Use sdk-fs for this connector.
* collection - This attribute is used to unpack temporary files. The default value is crawler-fs
* logging-config - Specifies the file used for configuring logging options; it must be formatted as a log4j XML string.
* classname - Java class name for the connector. The value to use this connector must be plugin:filesystem.plugin@filesystem.







 Configuring the filesystem crawl seed 

The following values can be configured for the filesystem crawl seed file. To set these values, open the file config/seeds/filesystem-seed.conf and specify the following values specific to your use cases:



* url - Newline-separated list of files and folders to crawl. UNIX users can use a path such as /usr/local/.

The URLs must start with sdk-fs://. So to crawl, for example, /home/watson/mydocs, the value of this URL is sdk-fs:///home/watson/mydocs. The third / in the URL is necessary.

The filesystems used by Linux, UNIX, and UNIX-like computer systems can contain special types of files, such as block and character device nodes and files that represent named pipes, which cannot be crawled because they do not contain data, but serve as device or I/O access points. Attempting to crawl such files generate errors during the crawl. To avoid such errors, exclude the /dev directory in any top-level crawl on a Linux, UNIX, or UNIX-like filesystem. If present on the system that you are crawling, also exclude temporary system directories, such as /proc, /sys, and /tmp, that contain transient files and system information.
Create the directory by uploading an empty folder from your desktop to the bucket. In the console, navigate to your bucket. Select Resource List > Storage > sdh_cos_k8 > sdh-cos-vora-bucket and click Upload.
5. Select the empty folder and name it checkpoints.

The first time you upload, you may be required to install the free tool, Aspera Connect. Another option is to create the directory [by using the aws command line tool](https://aws.amazon.com/cli/). Using this tool is not covered in this topic.





 Creating the service instance and credentials for accessing the bucket 



1. Follow the steps under [Service credentials](https://cloud.ibm.com/docs/services/cloud-object-storage/iam?topic=cloud-object-storage-service-credentials) to create the service credentials for accessing the bucket that is handed over to the SAP Data Hub installation script. Make sure that you select the Writer role and click Include HMAC Credential.

The credential name in this example is sdhOScred. Choose a name that fits your needs when creating your service credentials.

After the service credentials have been created, click View credentials and note the values of access_key_id and secret_access_key. See below for an example.



... "cos_hmac_keys": { "access_key_id": "383cf3", "secret_access_key": "a24****************0f9" },

 Locating the Installation Dialog parameters {: rhos-find-parameters}

During installation of SAP Data Hub, you're prompted to:

Enable Vora checkpoint store? (yes/no).

If you're setting up a test environment, you can select yes or no. However, for a production environment, select yes. You're next asked for the following parameters:

 {: screen}
Please provide the following parameters for Vora's checkpoint store
[root@sdb192 ] ln -s /opt/rh/SAP/lib64/compat-sap-c++.so /usr/sap/lib/libstdc++.so.6







 Downloading your SAP software 

Log in to the [SAP Support Portal)](https://support.sap.com/en/index.html), click Download Software, and download the required DVDs to a local share drive. Transfer the files to your provisioned server. Another option is to download the [SAP Software Download Manager)](https://support.sap.com/en/my-support/software-downloads.htmlsection_995042677), install it on your target server and directly download the DVD images to the server.





 Preparing for SAP's SWPM GUI 

Depending on your network bandwidth and latency, you might want to run the SAP Software Provisioning Manager (SWPM) graphical user interface (GUI) remotely in a virtual network computing (VNC) session. Another option is to have the GUI running locally and connect to SWPM on the target machine. Use the [SWPM documentation)](https://wiki.scn.sap.com/wiki/display/SL/Software+Provisioning+Manager+1.0+and+2.0) if you decide to run the GUI locally.

The following steps outline running the SWPM GUI remotely in a virtual network computing (VNC) session. This option installs a VNC server, which might not be inline with hardening your operating system; ensure that you are meeting your security measures. Refer to [VNC documentation](http://searchnetworking.techtarget.com/definition/virtual-network-computing) for an overview on its functions if you are not familiar with it.



1. Use the following commands to install a VNC server.



[root@sdb192 ] yum install tigervnc-server



2. Use the following command to install the X11 window manager, twm, which is included in the Linux distribution.



[root@sdb192 ] yum install twm



3.
Settings 

Use settings to add more configurations required by external applications to access App Configuration resources. By default, the Settings page displays Allowlist Domains.



  Allowlisting domains 

The Allowlist Domains is used for enabling Cross-Origin Resource Sharing (CORS) when using client-side browser SDKs' to access App Configuration resources by external applications. CORS is a specification, which enables controlled access to App Configuration resources from resources that are located outside of a domain.



  Adding domains to allowlist 

To add domains to allowlist, complete these steps:



1.  In the App Configuration console, click Settings.

Zoom

![List of additional settings](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-settings-default.png)

Figure 1. List of additional settings
2.  Click Edit. The side panel opens with fields for adding URLs to be allowlisted.
3.  Enter the URL of the domain that needs to be added to Allowlisted Domains and click Add +. Repeat this step to add more URLs.

Zoom

![Allowlisted domains](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-settings-allowlist.png)

Figure 2. List of allowlisted domains
4.  Click Save.



You can also modify the URL of the domains by using the Edit option.

If you are using JavaScript or React SDK, you need to allowlist your domains using the above steps or using App Configuration API. For more information, see [here](https://cloud.ibm.com/apidocs/app-configurationupdate-originconfigs).
* sed (stream editor)







 Obtaining the crawler plug-in SDK package 



1. Log in to your Discovery cluster.
2. Enter the following command to obtain your crawler pod name:

oc get pods | grep crawler

The following example shows sample output.

wd-discovery-crawler-57985fc5cf-rxk89 1/1 Running 0 85m
3. Enter the following command to obtain the SDK package name, replacing {crawler-pod-name} with the crawler pod name that you obtained in step 2:

oc exec {crawler-pod-name} -- ls -l /opt/ibm/wex/zing/resources/ | grep wd-crawler-plugin-sdk

The following example shows sample output.

-rw-r--r--. 1 dadmin dadmin 35575 Oct 1 16:51 wd-crawler-plugin-sdk-${build-version}.zip
4. Enter the following command to copy the SDK package to the host server, replacing {build-version} with the build version number from the previous step:

oc cp {crawler-pod-name}:/opt/ibm/wex/zing/resources/wd-crawler-plugin-sdk-${build-version}.zip wd-crawler-plugin-sdk.zip
5. If necessary, copy the SDK package to the development server.







 Building a crawler plug-in package 



1. Extract the SDK compressed file.
2. Implement the plug-in logic in src/. Ensure that the dependency is written in build.gradle.
3. Enter gradle packageCrawlerPlugin to create the plug-in package. The package is generated as build/distributed/wd-crawler-plugin-sample.zip.
[Create a Speech to Text service instance](https://cloud.ibm.com/catalog/services/speech_to_text).
2. When the service instance is created, service credentials are automatically created for you.







 Installing the Speech to Text package 

After you have an Speech to Text service instance, install the Speech to Text package into your namespace.





 Installing from the Cloud Functions CLI 

Install the Speech to Text package from the CLI. Be sure to [install the Cloud Functions plug-in for the IBM Cloud CLI](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-cli_install) first.

To install the Speech to Text package, follow these steps.



1. Clone the Speech to Text package repo.

git clone https://github.com/watson-developer-cloud/openwhisk-sdk
2. Deploy the package.

ibmcloud fn deploy -m openwhisk-sdk/packages/speech-to-text-v1/manifest.yaml
3. Verify that the package is added to your package list.

ibmcloud fn package list

Example output

packages
/myOrg_mySpace/speech-to-text-v1 private
4. Bind the credentials from the Speech to Text instance you created to the package.

ibmcloud fn service bind speech_to_text speech-to-text-v1

Depending on the region where you created the service instance, the service instance might be named differently because it is an IAM service. If the command fails, use the following service name for the bind command.

ibmcloud fn service bind speech-to-text speech-to-text-v1

Example output

Credentials 'Credentials-1' from 'speech_to_text' service instance 'Watson Speech to Text' bound to 'speech-to-text-v1'.
5. Verify that the package is configured with your Speech to Text service instance credentials.

ibmcloud fn package get speech-to-text-v1 parameters

Example output
The appConfigClient, once initialised, can be obtained across modules using AppConfiguration.getInstance().

Where:



* region: Region where the service instance is created. Use AppConfiguration.REGION_US_SOUTH for Dallas, AppConfiguration.REGION_US_EAST for Washington DC, AppConfiguration.REGION_EU_GB for London, and AppConfiguration.REGION_AU_SYD for Sydney.
* guid: Instance ID of the App Configuration service. Get it from the service credentials section of the App Configuration service dashboard.
* apikey: API key of the App Configuration service. Get it from the service credentials section of the App Configuration service dashboard.
* collectionId: ID of the collection created in App Configuration service instance under the Collections section.
* environmentId: ID of the environment created in App Configuration service instance under the Environments section.





Ensure to create the service credentials of the role Client SDK for using with the JavaScript SDK. API key of the Client SDK role has minimal access permissions that are suitable to use in browser based applications.



 Examples for using feature and property-related APIs 

See the following examples for using the feature-related APIs.



 Get single feature 

const feature = client.getFeature('feature_id')

if(feature) {

console.log('data', feature);
console.log(Feature Name ${feature.getFeatureName()} );
console.log(Feature Id ${feature.getFeatureId()} );
console.log(Feature Type ${feature.getFeatureDataType()} );
console.log(Feature is enabled ${feature.isEnabled()} );
}





 Get all features 

const features = client.getFeatures();

const feature = features["feature_id"];

if(feature) {
console.log(Feature Name ${feature.getFeatureName()});
console.log(Feature Id ${feature.getFeatureId()});
The GitHub tool integration supports the following configuration parameters that you can use with the [Toolchain HTTP API and SDKs](https://cloud.ibm.com/apidocs/toolchain) when you [create](https://cloud.ibm.com/apidocs/toolchaincreate-tool), [read](https://cloud.ibm.com/apidocs/toolchainget-tool-by-id), and [update](https://cloud.ibm.com/apidocs/toolchainupdate-tool) tool integrations.

You must specify the tool_type_id property in the request body with the githubconsolidated value.



Table 1. GitHub tool integration parameters

 Parameter Usage Type Terraform argument Description 

 api_root_url optional, updatable String api_root_url The URL of the API root for the GitHub server. 
 api_token optional, updatable Password api_token The Personal Access Token (PAT). This parameter is required only if the auth_type is set to pat, otherwise it is ignored. 
 auth_type optional, updatable, Default: oauth String auth_type Set the authentication method to use to access the Git provider. 
 auto_init optional, immutable, Default: false Boolean auto_init Set this value to true to initialize this repo with a readme file. This parameter is used only when you create a repo. 
 default_branch optional, updatable String default_branch The default branch of the Git repo. 
 enable_traceability optional, updatable, Default: false Boolean enable_traceability Set this value to true to track the deployment of code changes by creating tags, labels and comments on commits, pull requests, and referenced issues. 
 git_id optional, immutable String git_id Set this value to github for github.com, or to the GUID of a custom GitHub Enterprise server.