vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Building a Cloud Pak for Data custom connector 

Discovery provides connectors to many popular data sources, as described in [Configuring Cloud Pak for Data data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-types). If you need to connect to a different data source, you can write and deploy a custom connector.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments.

Any custom code that is used with IBM Watson® Discovery is the responsibility of the developer and is not covered by IBM support.

Example code and configuration files for a basic custom connector are included.

Related topics:



* [Developing custom Cloud Pak for Data connector code](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-dev)
* [Assembling, compiling, and packaging a custom Cloud Pak for Data connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-assemble)
* [Installing and uninstalling a custom Cloud Pak for Data connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install-connector)
* [Using a custom Cloud Pak for Data connector from the Discovery user interface](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-ccs-tooling)





 Custom connector requirements 

A custom connector is a component that uses the SDK and crawler framework that is documented here to connect to and crawl a specific data source. Custom connectors have the same general requirements as provided connectors. For more information, see [Data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements).

Before you implement a custom connector, you need to know the following information about the data source:



* The data source's network location (server name or address, including port, or URL, including port)
* The data source's authentication method and security credentials
Naming 



 Consistency of names 

The names of classes, methods, properties, and parameters MUST be consistent across all SDKs for a service and SHOULD be consistent with the names in the underlying API.





 Package names 

The name of the SDK's installable package MUST be descriptive of the set of services that are supported by the SDK and be consistent across all SDKs, to the extent allowed by the [package managers](https://cloud.ibm.com/docs/sdk-handbook?topic=sdk-handbook-devtoolsdevtools-pkg-mgmt).

For IBM Cloud SDKs, the SDK package name SHOULD be directly derived from the category of the SDK services as displayed in the [IBM Cloud API & SDK reference library](https://cloud.ibm.com/docs?tab=api-docs). Under this pattern, each service SHOULD have a corresponding module that contains its API operations. For example, the [Platform Services Node SDK](https://github.com/IBM/platform-services-node-sdk) includes the services found in the [Platform Services](https://cloud.ibm.com/docs?tab=api-docs&category=platform_services) category as separate modules, such as the [Global Search](https://cloud.ibm.com/apidocs/search?code=node) service that can be imported into your Node.js application using an import path of @ibm-cloud/platform-services/global-search/v2. These multi-service SDKs have the same structure as a single-service SDK, but lower the barrier to entry for user adoption and exploration of other services while potentially providing shared utilities across similar services.





 Classes, structs, and types 

Names for classes, structs, or types SHOULD be simple, descriptive, and meaningful to developers. Such names should generally be nouns or noun phrases.

For example:



* Use Trait rather than TraitTreeNode
* Use Classifier rather than GetClassifiersTopLevelBrief







 Methods 

Names for methods or functions SHOULD follow the convention of <verb>_<noun>, where the noun is typically the resource type

[1].
The HSM client image can be built with a Docker file similar to the following:



* The following Docker file is for IBM Cloud HSM. If you are not using IBM Cloud HSM, you need to build your own Docker file.
* You should be aware that this Docker file automatically accepts the Gemalto client license.
* Note that the 64 folder inside the Docker file is required for installing the HSM client.



FROM registry.access.redhat.com/ubi8/ubi-minimal as builder

 This directory contains the installation files for gemalto/luna client
COPY 64 64

RUN microdnf install -y 
gcc 
gcc-c++ 
openssh-clients 
bind-utils 
iputils 
&& cd 64 && 
 NOTE we are accepting the license for installing gemalto client here
 please take a look at the license before moving forward
echo "y" | ./install.sh -p sa

 Final image 

FROM registry.access.redhat.com/ubi8/ubi-minimal

 Copy the library files from builder
COPY --from=builder /usr/lib/libCryptoki2_64.so /usr/lib/libCryptoki2_64.so
COPY --from=builder /usr/lib/libCryptoki2_64.so.2 /usr/lib/libCryptoki2_64.so.2
COPY --from=builder /usr/lib/libCryptoki2_64.so.6.3.0 /usr/lib/libCryptoki2_64.so.6.3.0
Show more

Now, run the following command to build the Docker image:

docker build -t hsm-client:v1 -f Dockerfile .





 Step three: Push the Docker image to your container registry 

After the image is built, the next step is to push the image to your Docker registry (for example, Docker Hub). The commands look similar to:
If the route doesn't belong to the current space, switch to the space or org that it belongs to by entering the following command:

ibmcloud cf target -o org_name -s space_name
3. Delete the app route by entering the following command:

ibmcloud cf delete-route domain_name -n host_name

For example:

ibmcloud cf delete-route cf.cloud.ibm.com -n app001







 Can't retrieve spaces in the org 

You can't create an app or a service if your current organization doesn't have a space that is associated with it.

 What’s happening 

When you try to create an app in IBM Cloud, you see the following error message:

BXNUI0515E: The spaces in the org weren't retrieved. Either a network connection problem occurred, or your current organization does not have a space associated with it.

 Why it’s happening 

This error often occurs the first time that you try to create an app or a service from the catalog when a space isn't created yet.

 How to fix it 

Ensure that you created a space in your current organization. To create a space, use one of the following methods:



* From the menu bar, click Manage > Account, and select Cloud Foundry orgs. Click the name of the organization that you want to create the space in, and click Add a space.
* In the Cloud Foundry command-line interface, type cf create-space <space_name> -o <organization_name>.



Try again. If this message occurs again, go to the [IBM Cloud status](https://cloud.ibm.com/status) page to check whether a service or component has an issue.





 Can't perform requested actions 

You can't complete actions without appropriate access authority.

 What’s happening 

When you try to perform actions for a service instance or an app instance, you can't complete the requested actions and see one of the following error messages:

BXNUI0514E: You are not a developer for any of the spaces in the <orgName> organization.
vzpkgcache Update a set of preinstalled container archives after new template installation. 







 Supplementary tools 

Supplementary tools are used for miscellaneous tasks in the hardware node and container context.



Table 6. Supplemental tools

 Supplementary Tools 

 vzup2date Utility to update your Virtuozzo software and templates. 
 vzup2date-mirror Utility to create local mirrors of the Virtuozzo official repository. 
 vzfsutil Utility for the VZFS optimization and consistency checking. 
 vzcache Utility to gain extra disk space by caching the files identical in different containers. 
 vzsveinstall Utility to create the Service Container on the Hardware Node. 
 vzsveupgrade Utility to update the packages inside the Service Container. 
 vzps Utility working as the standard ps and htop utilities, with container-related functions added. 
 vztop Utility working as the standard ps and htop utilities, with container-related functions added. 
 vzsetxinetd Utility to switch some services between stand-alone mode and xinetddependent mode. 
 vzdqcheck Print file space current usage from quota’s point of view. 
 vzdqdump Utility to dump the Container user or group quota limits and grace times from the kernel or the quota file or for loading them to a quota file. 
 vzdqload Utility to dump the Container user or group quota limits and grace times from the kernel or the quota file or for loading them to a quota file. 
 vznetstat Utility that prints network traffic usage statistic by containers. 
 vzcpucheck Utility for checking CPU usage by containers. 
 vzmemcheck Utility for checking the hardware node and container current memory parameters. 
 vzcalc Utility to calculate resource usage by a container. 
 vzcheckovr Utility to check the current system overcommitment and safety of the total resource control settings. 
 vzstat Utility to monitor the hardware node and container resources consumption in real time. 
 vzpid Utility that prints container ID that the process belongs to. 
 vzsplit Utility to generate container configuration file sample, “splitting” the hardware node into equal parts. 
 vzcfgscale Utility to scale the container configuration.
Using the Node.js SDK 

The IBM Analytics Engine Node.js SDK allows you to interact programmatically with the IBM Analytics Engine service API for serverless instances.



 Installing the SDK 

You can install the IBM Analytics Engine Node.js SDK using the Node package manager (npm).

Type the following command into a command line:

npm install iaesdk

You can find the source code in GitHub. See [ibm-iae-node-sdk](https://github.com/ibm/ibm-iae-node-sdk/). The iaesdk library provides complete access to the IBM Analytics Engine API.

To run the Node.js SDK, you need Node 4.x+. You need to provide the service endpoints and the API key when you create a IBM Analytics Engine service resource or a low-level client.

The service instance ID is also referred to as a instance GUID. You can retrieve the service instance ID when you create service credentials or through the CLI. See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless).





 Code samples using iaesdk 

Getting started with the Node.js SDK after you have installed it, involves sourcing credentials to the IBM Analytics Engine service, invoking the service and then issuing different cluster commands as shown in the following sample code snippets.

In addition to the sample code snippets in this topic, you can work with Node.js code samples from the [IBM Analytics Engine V3 API reference](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3?code=nodeintroduction).

The following code samples show how to:



* Authenticate to the IBM Analytics Engine service and create a service client:

const IbmAnalyticsEngineApiV3 = require('iaesdk/ibm-analytics-engine-api/v3');
const { IamAuthenticator } = require('iaesdk/auth');
Furthermore, multiple instances of the class are created (Step 2) and the method initState() is called once for each instance (Step 3). The reason every UDF results in a separate class loader is to allow different UDFs to use different versions of the same class (or library).

At runtime (OperatorGraph.execute() API), the UDF class is not loaded again because the class is loaded during extractor instantiation. The Java method that implements the UDF is invoked when necessary to compute pieces of the output. When the extractor is used within a single thread, that means anywhere from zero to possibly many invocations for each input document exist (and most likely multiple invocations throughout the life of the Operator Graph object). When the extractor is shared across multiple threads, different threads can hit the method around the same time (with different inputs).

If you require a specific part of the UDF code to be evaluated exactly once, for example, to initialize data structures needed by the UDF method across invocations, that code should not be placed in the UDF evaluation method, since that method is most likely executed multiple times throughout the life of the extractor (like the OperatorGraph object), and can be hit almost simultaneously when the extractor is shared across multiple threads. Instead:



1. For scalar UDFs, follow standard Java programming principles. For example, place the code in a static block.
2. For table UDFs, place the code in the initState() method, or follow standard Java programming principles. For example, place the code in a static block.



When doing so, remember that the class might be loaded multiple times during extractor compilation, instantiation and validation, as explained in Steps 1-3 above. If the code that initializes the UDF is placed in a static block, that code is executed each time that the class is loaded (potentially multiple times), therefore, introducing an overhead during compilation and operator graph instantiation. If the overhead is large, follow these best practices:



* To minimize overhead during compilation time, the best practice is to place compile-time heavy resources like large dictionaries or UDFs with large initialization time in a separate AQL module and export them.
In this case that is Org1, so select Org1 MSP from the drop down list. Make this organization an operator. Do not add any ordering service organizations here. Note that during the creation of a channel, all organizations that are added must already be members of the consortium.

The update policy determines how many organizations must sign channel update requests before the channel configuration can be changed. Because we only have one organization, we only have one option, 1 out of 1. But as more organizations are added to the channel, you will have the option to change the update policy to 1 out of 2, or 2 out of 2, and so on.

Next, specify the organization that is signing the channel creation request. This is, once again, the Org1 MSP. And also sign the request using the Org1 MSP Admin identity you created. Then review the channel creation request you're submitting. If it looks good to you, click Create Channel.

You should see a pending tile on the channel page. This is because you haven’t joined a peer to this channel. So click the pending tile and select the peer you want to join to the channel. In this case that’s the only peer we have, Peer Org1. You will also see that this peer is being made an anchor peer by default. Anchor peers bootstrap the communication between peers in different orgs, and are important for enabling features like Service Discovery, so leave this option enabled. Note also the state database of your peer, as every peer in this channel should use the same state database type to ensure consistency between ledgers. Note that by default CouchDB is selected for all peers.

After you have joined the channel you can inspect it. You’ll see the block height of one. That’s the config block that was created during the create channel flow. On the channel details tab you’ll see nodes and channel members.

To initiate a channel update, click the update button. This will allow you to edit many of the fields you specified when creating the channel, and to perform important tasks like adding channel members. The advanced configuration tab allows you to edit features like capability levels and ordering service consenters that were not edited during channel create.
Or, plan to deploy your instance by using AD/DNS VMs within your VMware cluster, which can be backed up by Veeam.

Beginning with VMware vCenter 6.5u2, VMware supports the backup of the vCenter Postgres database by using image-based backups. These backups use integrated suspend and resume scripts for the database during the backup window to ensure database integrity. If you want to use this method, choose to use Veeam to back up your vCenter Server and PSC. Use this service instead of using file-based backups. If you do so, you must use the Veeam quiesce feature to ensure database integrity.





 Restoring from backup 

Review the following special considerations when you restore your management backups.



* For vCenter and PSC, VMware provides an installer that can deploy a new virtual appliance and restore the configuration from backup.
* When you restore an appliance from backup, the installer detects the type of appliance (vCenter Server with embedded PSC) based on the backup information you provide.
* Because you deploy directly to one of your hosts, you might not be able to deploy to a distributed switch or port group. Create a temporary standard switch and port group for deploying the recovered appliances. Then, migrate one of your vmnics temporarily to this switch to provide network connectivity for your VMs. After deployment, you can migrate the VMs to the distributed port group and return the vmnic to the dvSwitch.
* For NSX, you might need to redeploy your NSX Manager and controllers before you restore the configuration from backup.
* Ensure that you familiarize yourself with the VMware considerations and limitations for vCenter backup and restore.







 Summary 

With proper planning, you can ensure that your VMware instance can suffer the loss of its management components and recover successfully. Ensure to regularly monitor the success of your backup jobs and availability of your backup data. Ensure to test your backup and restore plan regularly, for both your management infrastructure and your workloads.





 Related links
The App ID SDK creates a code verifier for the authorization request, which is the plain text version of the code challenge. Along with the authorization request, the client sends the code challenge and the challenge method that is used to encode the challenge.
3. The authentication flow is started by App ID in a new window.
4. The user chooses an identity provider to authenticate with and completes the sign-in process.
5. The App ID SDK on the application receives the grant code
6. The SDK then makes an XHR request to the App ID token endpoint along with the grant code and the code verifier to obtain access and identity tokens.



Are you using SSO for Cloud Directory? You can use the App ID client SDK to automatically obtain a new pair of tokens without requiring that the user explicitly sign-in. For more information, see [Silent login](https://cloud.ibm.com/docs/appid?topic=appid-single-pagespa-silent-login).



 Why Authorization Code + PKCE? 

Due to their nature, single-page applications are unable to store secrets securely. The Authorization + PKCE flow, a variation of the OAuth 2.0 Authorization Code flow, uses a one-time code verifier and challenge instead of a secret to address the concern. The verifier and the challenge ensure that the entity that calls the authorization and token endpoints are the same. This process means that an attacker would not be able to request tokens from the authorization server without also knowing the code verifier.

Although the Implicit flow is the current industry standard, several security flaws can be found. For example, the Implicit flow uses redirect URIs to obtain tokens, which the Authorization + PKCE flow circumvents by using an XHR request instead. Because of the security flaws, the Implicit flow is no longer recommended or considered safe due to the following reasons:



* The Implicit flow returns tokens as part of the URL - as a query parameter or in the hash fragment. By doing so, the tokens can be intercepted and tokens can be accessed.
Create a baseline 

Creating a baseline is an effective approach for either:



* Understanding an existing database instance performance to enable a mapping to requirements when migrating that instance to IBM Cloud.
* Capturing the performance of a new database instance in IBM Cloud.



In this approach, performance counters from Windows Performance Monitor capture SQL Server wait statistics to better understand current performance and potential bottlenecks of the database instance. Key counters include; CPU, memory, IOPS, throughput, and latency. The capture should include workload activities at peak times. Peak hours include not only business day user workload, but also other high load activities including end-of-day processing, and end-of-quarter processing. Analysis of the counters allow the optimal selection of compute and storage for the virtual server that hosts the SAL Server instance.



 Compute 

In IBM Cloud, a fundamental sizing concept is that you only want to provision the compute the application needs and then plan to scale up or down as the business requires. This means that you need to take advantage of as much of the virtual servers resources as possible, therefore, the virtual server vCPU sizing should be selected to keep the average CPU as high as possible without impacting the workload. Ideally, try to aim for using 80% of your vCPU on average, and allowing peaks above 90%, but not reaching 100% for any sustained period of time.

The following Windows Performance Monitor counters can help validate the CPU health of a SQL Server virtual server:



* \Processor Information(_Total)% Processor Time.
* \Process(sqlservr)% Processor Time.



Memory should be baselined and include both memory used by the operating system as well as the memory used internally by SQL Server. The following Windows Performance Monitoring counters can help validate the memory health of a SQL Server virtual server:



* \Memory\Available MBytes
* \SQLServer:Memory Manager\Target Server Memory (KB).
* \SQLServer:Memory Manager\Total Server Memory (KB).
* \SQLServer:Buffer Manager\Lazy writes/sec.
* \SQLServer:Buffer Manager\Page life expectancy.







 Storage 

The performance of a SQL Server instance is heavily dependent on storage performance, which is measured by IOPS and throughput counters.
* At the end of the documented process, your vCenter Server instance is running vSphere 7.0 Update 1c with N-VDS distributed switches. This configuration is different than the currently supported [Software BOM for vCenter Server instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_bomvc_bom-software), which is vSphere 7.0 Update 3c with Distributed vSwitch 7.0.0.
* For more information about the migration of N-VDS to VDS switches for vSphere 7.0 or later and NSX-T Data Center 3.0 and later, see [Migrate host switch to vSphere Distributed Switch](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-1039A36F-F55E-4A0A-B6C6-2C383F4A716D.html). Currently, this procedure is not verified on a VMware Solutions vCenter Server instance.
* IBM Cloud is undertaking an assessment of the N-VDS to VDS conversion and the required changes to the automation database to allow this in-place upgrade.
* Currently, Day 2 automation workflows such as add host or add cluster, are not tested against VMware Solutions vCenter Server instances that are upgraded from vSphere 6.7 to 7 and still using N-VDS distributed switches. Customers must assume that this automation might fail and that if this automation is required then the lift and shift migration approach used. If this automation is not needed, you can use the upgrade process that is documented.
* A workaround for the add nodes and add cluster features is to use the VMware vSphere offering. For more information, see [VMware vSphere overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverview). You must complete a number of manual tasks after the automated deployment.