vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Enterprise PKCS #11





 Description Binds to EP11 m_DecryptSingle. 
 Parameters message DecryptSingleRequest { bytes Key = 1; Mechanism Mech = 2; bytes Ciphered = 3; } message DecryptSingleResponse { bytes Plain = 4; } 
 Return values Wraps EP11 error into message Grep11Error. 



Code snippets



* Golang code snippet

// Generate 16 bytes of random data for the initialization vector
GenerateRandomRequest := &pb.GenerateRandomRequest{
Len: (uint64)(ep11.AES_BLOCK_SIZE),
}
GenerateRandomResponse, err := cryptoClient.GenerateRandom(context.Background(), GenerateRandomRequest)
if err != nil {
return nil, fmt.Errorf("GenerateRandom error: %s", err)
}
iv := GenerateRandomResponse.Rnd[:ep11.AES_BLOCK_SIZE]
fmt.Println("Generated IV")

DecryptSingleRequest := &pb.DecryptSingleRequest {
Key: GenerateKeyResponse.KeyBytes,
Mech: &pb.Mechanism{Mechanism: ep11.CKM_AES_CBC_PAD, Parameter: util.SetMechParm(iv)},
Ciphered: EncryptSingleResponse.Ciphered, // encrypted data from a previous encrypt operation
}

DecryptSingleResponse, err := cryptoClient.DecryptSingle(context.Background(), DecryptSingleRequest)
Show more
* JavaScript code snippet

client.DecryptSingle({
Mech: {
Mechanism: ep11.CKM_AES_CBC_PAD,
ParameterB: iv
},
Key: bobDerived.NewKey,
Ciphered: ciphertext
repeated google.protobuf.Any details = 3;
}

In the error message,



* code includes the status code, which needs to be an enumerated-type (enum) value of the google.rpc.Code field.
* message includes a developer-facing error message in English. Any user-facing error message needs to be localized and sent in the google.rpc.Status.details field, or localized by the user.
* details lists messages that carry the error details. A common set of message types is available for the API to use.



GREP11 uses the Detail field to attach extra error code information.

message Grep11Error {
uint64 Code = 1;
string Detail = 2;
bool Retry = 3;
}

The Code field can be cast to the CK_RV value in PKCS #11. This field contains the error codes that are defined by the

[PKCS #11 specification](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959729) or the vendor extensions that are defined by EP11. EP11 uses only a subset of return values that PKCS 11 defines. For more information, see the 10.1.6 Return values section in

[Enterprise PKCS #11 Library structure](https://www.ibm.com/security/cryptocards/pciecc4/library).

An [example](https://github.com/IBM-Cloud/hpcs-grep11-go/blob/master/examples/server_test.goL863) in Golang that deals with errors is available.





 GREP11 function list 

PKCS #11 functions that are marked with an asterisk (*) in the table are implemented by EP11 over gRPC. Others are not implemented.



Table 1. Describes the implemented functions in EP11 over gRPC

 PKCS #11 Enterprise PKCS #11 Enterprise PKCS #11 over gRPC Description
When you generate a key, the key is added to the list.







 Step 2. Send logs 

To send logs, run the following cURL command:

curl "ENDPOINT/logs/ingest?QUERY_PARAMETERS" -u INGESTION_KEY: --header "Content-Type: application/json; charset=UTF-8" -d "LOG_LINES"

Where



* ENDPOINT represents the entry point to the service. Each region has a different URL. To get the endpoint for a location, see [Ingestion endpoints](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-endpointsendpoints_ingestion).
* QUERY_PARAMETERS are parameters that define the filtering criteria that are applied to the ingestion request.
* LOG_LINES describe the set of log lines that you want to send. It is defined as an array of objects.
* INGESTION_KEY is the key that you created in the previous step.



The following table lists the query parameters:



Query parameters

 Query parameter Type Status Description 

 hostname string required Host name of the source. 
 mac string optional The network mac address of the host computer. 
 ip string optional The local IP address of the host computer. 
 now date-time optional The source UNIX timestamp in milliseconds at the time of the request. Used to calculate time drift. 
 tags string optional Tags that are used to dynamically group hosts. 



The following table lists the data that is required per log line:



Line object fields

 Parameters Type Description 

 timestamp UNIX timestamp, including milliseconds, when the log entry was recorded. 
 line string Text of the log line. 
 app string Name of the application that generates the log line. 
 level string Set a value for the level. For example, sample values for this parameter are INFO, WARNING, ERROR. 
 meta This field is reserved for custom information that is associated with a log line. To add metadata to an API call, specify the meta field under the lines object. Metadata can be viewed inside that line's context.
WebSocket 

The preinstalled /whisk.system/websocket package for IBM Cloud® Functions offers a convenient way to post messages to a WebSocket.

The package includes the following actions.



Table 1. WebSocket package entities

 Entity                          Type     Parameters        Description                                 

 /whisk.system/websocket         Package  uri               Utilities for communicating with WebSockets 
 /whisk.system/websocket/send    Action   uri, payload      Send the payload to the WebSocket URI.      



If you plan to send many messages to the same WebSocket URI, creating a package binding with the uri value is suggested. With binding, you don't need to specify the value each time that you use the send action.



  Send a message to a WebSocket 

The /whisk.system/websocket/send action sends a payload to a WebSocket URI. The parameters are as follows.



Table 2. WebSocket send action parameters

 Parameter  Description                                                                

 uri        The URI of the WebSocket server. For example, ws://mywebsockethost:80.     
 payload    The message to send to the WebSocket.
For example, if you specify purple unicorn in the Secret field, you might add code similar to this:

const jwt = require('jsonwebtoken');
...
const token = request.headers.authentication; // grab the "Authentication" header
try {
const decoded = jwt.verify(token, 'purple unicorn');
} catch(err) {
// error thrown if token is invalid
}





 Webhook request body 

The request body the webhook sends to the external service is a JSON object with the following structure:

{
"event": {
"name": "message_logged"
},
"payload": {
...
}
}

The payload object contains the event data to be logged. The structure of the payload object depends on the type of event.



 Message event payload 

For message_logged events, the payload object contains data about a message request sent to the assistant and the message response returned to the integration or client application. For more information about the fields that are part of message requests and responses, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-data-v2message).

The log webhook payload might include data that is not currently supported by the API (for example, data returned from actions skills). Any fields that are not defined in the API reference documentation are subject to change.
The request body the webhook sends to the external service is a JSON object with the following structure:

{
"event": {
"name": "{event_type}"
},
"payload": {
...
}
}

where {event_type} is either message_logged (for messages and responses) or cdr_logged (for CDR events).

The payload object contains the event data to be logged. The structure of the payload object depends on the type of event.



 Message event payload 

For message_logged events, the payload object contains data about a message request sent to the assistant and the message response returned to the integration or client application. For more information about the fields that are part of message requests and responses, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-v2message).

The log webhook payload might include data that is not currently supported by the API (for example, data returned from actions skills). Any fields that are not defined in the API reference documentation are subject to change.





 CDR event payload 

For cdr_logged events, the payload object contains data about a Call Detail Record (CDR) event that was handled by the phone integration. The structure of the payload object for a CDR event is as shown by this example:

{
"primary_phone_number": "+18005550123",
"global_session_id": "9caa8bad-aaa8-4a5a-a4b5-62bccc703d15",
"failure_occurred": false,
"transfer_occurred": false,
"active_calls": 0,
"warnings_and_errors": [
{
"code": "CWSMR0033W",
"message": "CWSMR0033W: The inbound RTP audio stream jitter of 43 ms exceeds the maximum jitter threshold of 30 ms."
},
{
"code": "CWSMR0070W",
[semver](https://www.npmjs.com/package/semver) Semantic Versioning for Node.js. 
 [serialize-error](https://www.npmjs.com/package/serialize-error) Serialize an error into a plain object. 
 [serve-favicon](https://www.npmjs.com/package/serve-favicon) Node.js middleware for serving a favicon. 
 [socket.io](https://www.npmjs.com/package/socket.io) Socket.io enables real-time bidirectional event-based communication. 
 [socket.io-client](https://www.npmjs.com/package/socket.io-client) Realtime application framework for socket.io. 
 [superagent](https://www.npmjs.com/package/superagent) SuperAgent is a small progressive client-side HTTP request library, and Node.js module with the same API, sporting many high-level HTTP client features. 
 [swagger-tools](https://www.npmjs.com/package/swagger-tools) Package that provides various tools for integrating and interacting with Swagger. 
 [tmp](https://www.npmjs.com/package/tmp) A simple temporary file and directory creator for Node.js. 
 [ts-jest](https://www.npmjs.com/package/ts-jest) A TypeScript preprocessor with source map support for Jest that enables you use Jest to test projects written in TypeScript. 
 [twilio](https://www.npmjs.com/package/twilio) A wrapper for the Twilio API, related to voice, video, and messaging. 
 [underscore](https://www.npmjs.com/package/underscore) Underscore.js is a utility-belt library for JavaScript that supports the usual functional suspects (each, map, reduce, filter, and so on) without extending any core JavaScript objects.
Using the Kafka API 

Kafka provides a rich set of APIs and clients across a broad range of languages. APIs include core API, Streams API, and Connect API.



* Kafka's core API (Consumer, Producer, and Admin API)
Use to send and receive messages directly from one or more Kafka topics. The Kafka Admin client provides a simple interface through the Kafka API for managing Kafka resources. You can create, delete, and manage topics. You can also use the Admin client to manage consumer groups and configurations.
* Streams API
A higher-level stream processing API to easily consume, transform, and produce events between topics.
* Connect API
A framework that allows reusable or standard integrations to stream events into and out of external systems, such as databases.



The following table summarizes what you can use with Event Streams:



Table 1. Kafka client support in Standard, Enterprise, and Lite plans.

 Enterprise plan Standard plan Lite plan 

 Kafka version on cluster Kafka 3.3 Kafka 3.3 Kafka 3.3 
 Supported client versions Kafka 0.10.x, or later Kafka 0.10.x, or later Kafka 0.10.x, or later 
 Kafka Connect supported Yes Yes No 
 Kafka Streams supported Yes Yes No 
 ksqlDB supported Yes No No 
 Authentication requirements Client must support authentication by using the SASL Plain mechanism and use the Server Name Indication (SNI) extension to the TLSv1.2 protocol. Client must support authentication by using the SASL Plain mechanism and use the Server Name Indication (SNI) extension to the TLSv1.2 protocol. Client must support authentication by using the SASL Plain mechanism and use the Server Name Indication (SNI) extension to the TLSv1.2 protocol. 





 Choosing a Kafka client to use with Event Streams 

The official client for the Kafka API is written in Java, and as such contains the latest features and bug fixes.
The WebSocket interface 

To synthesize text to speech with the WebSocket interface of the IBM Watson® Text to Speech service, you first establish a connection with the service by calling its /v1/synthesize method. You then send the text to be synthesized to the service as a JSON text message over the connection. The service automatically closes the WebSocket connection when it finishes processing the request.

The synthesize request and response cycle includes the following steps:



1. [Open a connection](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-usingWebSocketWSopen).
2. [Send input text](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-usingWebSocketWSsend).
3. [Receive a response](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-usingWebSocketWSreceive).



The WebSocket interface accepts identical input and produces identical results as the GET and POST /v1/synthesize methods of the HTTP interface. In addition, the WebSocket interface also supports use of the SSML <mark> element to identify the location of user-specified markers in the audio. It can also return timing information for all strings of the input text. (The <mark> element and word timings are available only with the WebSocket interface.)



* For more information about obtaining word timings, see [Generating word timings](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-timing).
* For more information about the WebSocket interface and its parameters, see the [API & SDK reference](https://cloud.ibm.com/apidocs/text-to-speech).



The snippets of example code that follow are written in JavaScript and are based on the HTML5 WebSocket API.
Header name Header value 

 Content-Type application/json 



The service automatically sends an Authorization header with a JWT; you do not need to add one. If you want to handle authorization yourself, add your own authorization header and it will be used instead.



Your webhook details are saved automatically.





 Testing the webhook 

Do extensive testing of your webhook before you enable it for an assistant that is being used in a production environment.

Your assistant must have a skill added to it before your webhook can do anything useful. The webhook is triggered when a message is sent to your assistant to be processed by the associated skill.

If you enable the setting that returns an error when the webhook call fails, the processing of the assistant is halted entirely if the webhook encounters any issues. Take steps to test the process that you are calling on a regular basis so you will be alerted if the external service is down, and can take actions to prevent all of the incoming messages from failing to be recieved.

If you call an Cloud Functions web action, you can use the logging capability in Cloud Functions to help you troubleshoot your code. You can [download the command line interface](https://cloud.ibm.com/functions/learn/cli), and then enable logging with the [activation polling command](https://cloud.ibm.com/docs/cloud-functions-cli-plugin?topic=cloud-functions-cli-plugin-functions-clicli_activation_poll).





 Troubleshooting the webhook 

The following error codes can help you track down the cause of issues you might encounter. If you have a web chat integration, for example, you will know that your webhook has an issue if every test message you submit returns a message such as, There is an error with the message you just sent, but feel free to ask me something else. If this message is displayed, use a REST API tool, such as cURL, to send a test /message API request, so you can see the error code and full message that is returned.



Error code details

 Error code and message Description
Header example

 Header name Header value 

 Content-Type application/json 



The service automatically sends an Authorization header with a JWT; you do not need to add one. If you want to handle authorization yourself, add your own authorization header and it will be used instead.

After you save the header value, the string is replaced by asterisks and can't be viewed again.



Your webhook details are saved automatically.





 Testing the webhook 

Do extensive testing of your webhook before you enable it for an assistant that is being used in a production environment.

Your assistant must have a skill added to it before your webhook can do anything useful. The webhook is triggered when a message is sent to your assistant to be processed by the associated skill.

If you enable the setting that returns an error when the webhook call fails, the processing of the assistant is halted entirely if the webhook encounters any issues. Take steps to test the process that you are calling on a regular basis so you will be alerted if the external service is down, and can take actions to prevent all of the incoming messages from failing to be received.

If you call an Cloud Functions web action, you can use the logging capability in Cloud Functions to help you troubleshoot your code. You can [download the command line interface](https://cloud.ibm.com/functions/learn/cli), and then enable logging with the [activation polling command](https://cloud.ibm.com/docs/cloud-functions-cli-plugin?topic=cloud-functions-cli-plugin-functions-clicli_activation_poll).





 Troubleshooting the webhook 

The following error codes can help you track down the cause of issues you might encounter. If you have a web chat integration, for example, you will know that your webhook has an issue if every test message you submit returns a message such as There is an error with the message you just sent, but feel free to ask me something else. If this message is displayed, use a REST API tool, such as cURL, to send a test /message API request, so you can see the error code and full message that is returned.



Error code details

 Error code and message Description
const decoded = jwt.verify(token, 'purple unicorn');
} catch(err) {
// error thrown if token is invalid
}





 Request body 

It is useful to know the format of the request body of the premessage webhook so that your external code can process it.

The payload contains the request body of the /message (stateful and stateless) v2 API request. The event name message_received indicates that the request is generated by the premessage webhook. For more information about the message request body, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-data-v2message).

{
"payload" : { Copy of request body sent to /message }
"event": {
"name": "message_received"
}
}





 Response body 

The service that receives the POST request from the webhook must return a JSON object (Accept: application/json).

The response body must have the following structure:

{
"payload": {
...
}
}

The payload object in the response should contain the payload object that was received in the request body. Your code can modify property values in the message payload it received (for example, to update property values, or to add or remove context variables); but the message payload returned to the service must conform to the schema for a request to the message method. For more information, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-data-v2message).





 Example 1 

This example shows you how to check the language of the input text, and append the language info to the input text string.

In the premessage webhook configuration page, the following values are specified:



* URL: https://us-south.functions.appdomain.cloud/api/v1/web/e97d2516-5ce4-4fd9-9d05-acc3dd8ennn/default/check_language
* Secret: none
[kafkajs](https://www.npmjs.com/package/kafkajs) to connect to the Event Streams instance.
4. [Express](https://expressjs.com/) to enable a simple web server that allows interaction with the data.



There are five main files:



1. server.js: This runs the web server and communicates with Redis. It is getting regular calls from the front end to the /data route. It connects to Redis, downloads an object with the ID and position of every truck and returns that to the front end.
2. index.html: This is the one and only page of the front end application. When it loads, it shows a map of the US from the awesome [Open Street Map](https://www.openstreetmap.org/copyright) service. Then it is on a five-second loop where it makes an HTTP request to the backend (server.js), retrieves all the trucks and their positions and loops through them to plot their latest position on the map. Each truck has a map marker whose lat/long gets updated with every cycle.
3. producer.js: This takes in a location file and a truck_id from the script invocation and then loops through all the data points in the location file. Each data point is turned into a message with the truck ID and lat/long and posted once a second to Event Streams. For added fun, it picks a random starting point along the route and turns around and comes back when it reaches the end of the location file.
4. cloudantConsumer.js: This script creates a trucktracker database in Cloudant if it does not already exist and then polls Event Streams at regular intervals for more data. When data comes in, it gets written in bulk to Cloudant.
5. redisConsumer.js: This script polls Event Streams at regular intervals for data. When data comes in, it gets written into a Redis "hash" where the hash key is the truck ID and the value is the truck data (lat/long).
const decoded = jwt.verify(token, 'purple unicorn');
} catch(err) {
// error thrown if token is invalid
}





 Request body 

It is useful to know the format of the request body of the premessage webhook so that your external code can process it.

The payload contains the request body of the /message (stateful and stateless) v2 API request. The event name message_received indicates that the request is generated by the premessage webhook. For more information about the message request body, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-v2message).

{
"payload" : { Copy of request body sent to /message }
"event": {
"name": "message_received"
}
}





 Response body 

The service that receives the POST request from the webhook must return a JSON object (Accept: application/json).

The response body must have the following structure:

{
"payload": {
...
}
}

The payload object in the response should contain the payload object that was received in the request body. Your code can modify property values in the message payload it received (for example, to update property values, or to add or remove context variables); but the message payload returned to the service must conform to the schema for a request to the message method. For more information, see the [API reference](https://cloud.ibm.com/apidocs/assistant/assistant-v2message).





 Example 1 

This example shows you how to check the language of the input text, and append the language info to the input text string.

In the premessage webhook configuration page, the following values are specified:



* URL: https://us-south.functions.appdomain.cloud/api/v1/web/e97d2516-5ce4-4fd9-9d05-acc3dd8ennn/default/check_language
* Secret: none