vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Deploy an application from code on your local system with Code Engine.





* Video transcript

Hi, my name is JJ Asghar and I'm a developer advocate for IBM Cloud. Recently, you might have heard about Heroku changing their policy on free - the free tier. It's causing a challenge for a lot of developers out there so I want to take a handful of moments here to show you how you can migrate from Heroku to Code Engine from IBM Cloud in just a handful of steps. So let's go ahead and start playing around with it and see how quickly you can actually make your system work.

So first thing first, if you haven't seen it this is the actual line inside the official blog from Heroku, starting on November 28, 2022, we plan to stop free - offering the free product plans and plan on shutting down the free dynos and data services. We'll be sending out a series of email communications to affected users. This is challenging for a lot of beginner beginner web applications. I know for a fact personally I used Heroku when I first started back in the day so this is a pretty large hit for a lot of people and I want to show you how easy it is to convert from Heroku to Code Engine.

So first thing first, let's actually see a nice little application I've created. If I go ahead and bring up my application here, we have a nice little flask app. If you don't know what python is or flask is, it's a it's a format for python to be able to run an application on a standard port. So let's say we have this application; it says “Hello World!”. I've already deployed it and we can check out right here We have our amazing production app at heroku.com. It says “hello world” and we want to go ahead and change it. We wanted to update it so I'm going to go ahead and quickly update it to “Hello Moving from Heroku to Code Engine”.

Let’s go ahead and come out of it. git add .git commit -m “update hello line”.
I took the exact same code I did from Heroku. I created a new project and then I created an application and I just pushed it and it just worked so imagine what you can do with that for yourself. This shows the power that is Code Engine and on a free tier it is truly free - just like Heroku was or will be or was will won't be in the future. Code Engine is free forever, which is great and hopefully, it'll make your life a little easier.

Thanks so much for watching and if you have any questions, my Twitter handle is @jjasghar or you're more than welcome to email me at [awesome@ibm.com](mailto:awesome@ibm.com). My job is to be a personable nerd so never hesitate to reach out.

Thanks so much.

Bye y'all.







 Prerequisites 

Before you can get started with Code Engine, you need to set up your account and install the CLI.



* All Code Engine users are required to have a Pay-as-you-Go account.
* While you can use Code Engine through the console, the examples in this documentation focus on the command line. Therefore, you must install the Code Engine CLI.

ibmcloud plugin install code-engine

For more information, see [setting up the Code Engine CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-install-cli). For more information about the CLI, see [Code Engine CLI reference](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cli).







 Comparing Heroku and Code Engine terminology 

Before you get started with deploying apps in Code Engine, learn the basics about Code Engine. The following table describes some high-level terminology differences between Cloud Foundry and Code Engine.



Table 1. Terminology

 Heroku Code Engine Description 

 N/A Resource group and projects A grouping of workloads. The specific choice of which workload goes into each grouping is defined by the user. "Resource groups" are an IBM Cloud concept, while "projects" are Code Engine specific.
"arguments": ["{"library_set":{"action":"add","name":"customize_integration_custom_lib","script":{"source":"py_files","params":"https://s3.direct.<CHANGEME_REGION>.cloud-object-storage.appdomain.cloud","<CHANGEME_BUCKET_NAME>","libcalc.so","<CHANGEME_ACCESS_KEY>","<CHANGEME_SECRET_KEY>"]}}}"],
"conf": {
"spark.hadoop.fs.cos.dev-cos.endpoint":"https://s3.direct.<CHANGEME_REGION>.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.dev-cos.access.key":"<CHANGEME_ACCESS_KEY>",
"spark.hadoop.fs.cos.dev-cos.secret.key":"<CHANGEME_SECRET_KEY>",
"spark.submit.pyFiles":"cos://<CHANGEME_BUCKET_NAME>.dev-cos/customization_script.py"
}
}
}
Show more

Example of customization_script.py:

import cos_utils
import os
import sys

def customize(install_path, params):
print ("inside base install_misc_package(), override this to implement your own implementation.")
for param in params:
print(param)
endpoint = params[0]
bucket_name = params[1]
log_config_file = params[2]
access_key = params[3]
secret_key = params[4]
cos = cos_utils.get_cos_object(access_key, secret_key, endpoint)
[Shows the collection menu](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-collection-menu.png)

Figure 1. Collection menu
5. Click Create custom annotator.
6. Name your annotator, and then optionally add a description.
7. From the Annotator Type menu, select Dictionary, and then click Next.
8. Click the Import button, and then select the {name}.fdic.xml dictionary file that you want to import.
9. Click Save.







 Uploading dictionaries from Watson Explorer oneWEX 

You can import [dictionaries](https://www.ibm.com/support/knowledgecenter/en/SS8NLW_12.0.0/com.ibm.watson.wex.ee.doc/c_ce_adm_dict_ann.html) from IBM Watson Explorer oneWEX.



1. From Watson Explorer oneWEX, Version 12.0.0 or later modifications or fix packs, download the dictionary CSV file.



* Log in to the oneWEX administrator console.
* Open the Resource tab.
* Select dictionary enrichments, open the dictionary tab, and click the download icon. The dictionary is downloaded as a CSV file.



2. From your Discovery Content mining project, open the Content Mining application.
3. From the analysis view of your collection, click the Collections link in the breadcrumb to open the Create a collection for your analytics solutions page of the Content Mining application.
4. To create an annotator, click collection, and then select custom annotator from the list.

Zoom

!
ibm_headers = IbmAuthHelper.get_headers(URL, APIKEY, GUID)
sdclient = SdMonitorClient(sdc_url=URL, custom_headers=ibm_headers)

 Show the list of dashboards
ok, res = sdclient.get_dashboards()

if not ok:
print(res)
sys.exit(1)

for db in res['dashboards']:
print("%s [Team ID: %s] Name: %s, Charts: %d" % ( db['id'], db['teamId'], db['name'], len(db['widgets'] if 'widgets' in db else [])))
Show more





 List the dashboards that are available in a team 

The following code shows the structure of a Python script to list the dashboards available to a specific team.

!/usr/bin/env python3

import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))

from sdcclient import SdMonitorClient

 Parse arguments.
def usage():
print('usage: %s <endpoint-url> <sysdig_token>' % sys.argv[0])
print('endpoint-url: The endpoint URL that should point to IBM Cloud')
print('sysdig_token: Sysdig token for the team')
sys.exit(1)

if len(sys.argv) != 3:
usage()

URL = sys.argv[1]

 Set to the Sysdig token of the team
SYSDIG_TOKEN = sys.argv[2]

sdclient = SdMonitorClient(token=SYSDIG_TOKEN,sdc_url=URL)
Create an API 

See how easy it is to create an API by importing a YAML definition.

Complete the tutorial [Importing an API](https://www.ibm.com/support/knowledgecenter/SSMNED_v10/com.ibm.apic.apionprem.doc/tutorial_apionprem_import_api.html) in the extended API Connect V10 Reserved documentation.





 Next steps 

For more step-by-step examples on creating, securing, and testing APIs, see the additional [API Manager tutorials](https://www.ibm.com/support/knowledgecenter/SSMNED_v10cloud/com.ibm.apic.apionprem.doc/tapim_management_tutorials.html) in the extended API Connect V10 Reserved documentation.

If you're ready to focus on specific tasks, see the topics in the [Using API Connect Reserved](https://cloud.ibm.com/docs/apiconnect?topic=apiconnect-ri-user-over) section for more information on working with APIs and products, catalogs, event analytics, and the Developer Portal.





 About your personal data 

The API Connect service processes but doesn't store user data. You don't need to take any action to manage or delete your data.
* If you are using a private GitHub repository, provide your personal GitHub access token that you set up in [Setting up the IBM Cloud Schematics](https://cloud.ibm.com/docs/ibm-slurm?topic=ibm-slurm-setting-up-api) prerequisites in the x_github_token= "<github-api-token>" parameter. If you are using the public repository that is provided by IBM Cloud, you do not need to specify this parameter.
* Optional: Provide the tags if you want to filter resources by using the tag.



5. Run the Python script by using python3 <python-file-name> to create a Schematics workspace in the IBM Cloud.
6. You get a successful response if the parameters passed as part of the request are valid and you should be able to see the Schematics workspace that you created in the IBM Cloud console. If you don’t get a successful response, the error response contains the errors that you need to resolve. Resolve those errors and run the script until you are able to get a valid response and create a workspace.
7. Optional: If you want to add one of the variables such as base_name with the value slurm-test when you create a workspace, provide the values in the following parameters:



* workspace_variable_request_model = {}
* workspace_variable_request_model['name'] = 'base_name'
* workspace_variable_request_model['value'] = 'slurm-test'



8. If you want to update multiple values at the same time, follow the steps for [Updating variables with Schematics API](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-update-variables&interface=api).





 Example Python request 

 Create a Schematics workspace using Python API

import json, logging

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_schematics.schematics_v1 import SchematicsV1
Your app requires dependencies that are not included with the base Cloud Functions [Python runtime](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-runtimesopenwhisk_ref_python_environments) and your app code, even when it is compressed into a .zip file, is still larger than the maximum codeSize as described in the [Action Limits](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-limitslimits_actions). If so, you can install those dependencies into a custom Docker image and include your app code in the action/exec folder of the Docker skeleton. You can then specify your custom Docker image when you deploy your app in Cloud Functions. During deployment, you do not need to specify for your app code. Note that only public Docker images are supported.

Example command

ibmcloud fn action create <action_name> --docker <dockerhub_username>/<repo_name>:<tag_name>







 Packaging Python code 

The following sections provide tutorials for how to package your Python app for deployment with Cloud Functions.

Before you begin

Review [How do I package my Python app for deployment in Cloud Functions?](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-prephow_to_package_python).



 Packaging multiple Python files into a compressed file 

Package Python code and dependent modules in a compressed file. In this example, the source file that contains the entry point is __main__.py and the helper modules are in a file called helper.py.

Before you begin, [review the packages that are included with the Python runtime](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-runtimesopenwhisk_ref_python_environments) to see whether a dependency of your app is already included with the runtime. If your dependency is not included, you must package it with your app.



1. Create a test directory on your desktop.
2.
Building a Cloud Pak for Data custom crawler plug-in 

Discovery features the option to build your own crawler plug-in with a Java SDK. By using crawler plug-ins, you can now quickly develop relevant solutions for your use cases. You can download the SDK from your installed Discovery cluster. For more information, see [Obtaining the crawler plug-in SDK package](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-crawler-plugin-buildobtain-sdk).

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments.

Any custom code that you use with IBM Watson® Discovery is the responsibility of the developer; IBM Support does not cover any custom code that the developer creates.

The crawler plug-ins support the following functions:



* Update the metadata list of a crawled document
* Update the content of a crawled document
* Exclude a crawled document
* Reference crawler configurations, masking password values
* Show notice messages in the Discovery user interface
* Output log messages to the crawler pod console



However, the crawler plug-ins cannot support the following functions:



* Split a crawled document into multiple documents
* Combine content from multiple documents into a single document
* Modify access control lists





 Crawler plug-in requirements 

Make sure that the following items are installed on the development server that you plan to use to develop a crawler plug-in by using this SDK:



* Java SE Development Kit (JDK) 1.8 or higher
* [Gradle](https://gradle.org/install/)
* cURL
* sed (stream editor)







 Obtaining the crawler plug-in SDK package 



1. Log in to your Discovery cluster.
2. Enter the following command to obtain your crawler pod name:

oc get pods | grep crawler

The following example shows sample output.

wd-discovery-crawler-57985fc5cf-rxk89 1/1 Running 0 85m
3.
You can also [build your own actions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actions) or [web actions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-actions_web) to respond to the trigger.

For a list of the actions in the cloud-object-storage package, see [Available entities](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage_actions).

If you plan to use the client-get-signed-url action, you must [bind your service credentials](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage_sc_bind) to the package or to the client-get-signed-url action. If you do not plan to use this action, you can [set an IAM access policy](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_obstoragepkg_obstorage_rw_auth) for your Cloud Functions namespace.



 1. Installing the IBM Cloud Object Storage package 

You can install the cloud-object-storage package from the console or the CLI.



 Installing with the console 



1. In the Cloud Functions console, go to the [Create page](https://cloud.ibm.com/functions/create).
2. Select the namespace in which you want to install the IBM Cloud Object Storage package.
3. Click Install Packages.
4. Click the IBM Cloud Object Storage Package group, then click the IBM Cloud Object Storage Package.
5. In the Available Runtimes section, select either Node.JS or Python. Then, click Install.
6. After the package is installed, you are redirected to the Actions page and can search for your new package, which is named cloud-object-storage.
In this case, both the source code and package file are located in the same folder.



1. Create your source code by writing your code into a main.js file. For example, copy the following code example into a file called main.js.

function main(args) {
const oneLinerJoke = require('one-liner-joke');
let getRandomJoke = oneLinerJoke.getRandomJoke();

return {
headers: { 'Content-Type': 'text/plain;charset=utf-8' },
body: getRandomJoke
}
}

module.exports.main = main;
2. Create a package.json file that contains the required dependencies for your Function. For the previous code example, use the following contents for your package.json file.

{
"main": "main.js",
"dependencies" : {
"one-liner-joke" : "1.2.2"
}
}
3. Create your files as a Function in Code Engine. Both of previous files must be accessible in the repository. If they are in a private repository, create [private code repository access](https://cloud.ibm.com/docs/codeengine?topic=codeengine-code-repositories) and then provide that value with the --build-git-repo-secret option. If your files are located in a directory other than main, provide the path to the directory with the --build-context-dir option.

ibmcloud ce fn create -n nodejoke -runtime nodejs-18 --build-source GITHUB_DIR
4. Invoke your Function by pasting the provided URL into a web browser. Your browser displays a random joke!



For more information about the fn create command and its options, see [Create a Function](https://cloud.ibm.com/docs/codeengine?topic=codeengine-clicli-function-create).





 Including modules for a Python Function
If you use the name from the example, replace <name> with a unique identifier, such as cf-py-msx.

Attention: Do not use personal information in any app, container image, or Kubernetes resource names.

As the app is deployed, instructions for "Download, modify, and redeploy your app with the command line interface" are displayed.

c. From step 1 in the console instructions, click DOWNLOAD STARTER CODE.

d. Extract the .zip file and save its contents to your cf-py directory.



Your app code is ready to be containerized!





 Step 2: Create a Docker image with your app code 

Create a Dockerfile that includes your app code and the necessary configurations for your container. Then, build a Docker image from that Dockerfile and push it to your private image registry.



1. In the cf-py directory that you created in the previous lesson, create a Dockerfile, which is the basis for creating a container image. You can create the Dockerfile by using your preferred CLI editor or a text editor on your computer. The following example shows how to create a Dockerfile file with the nano editor.

nano Dockerfile
2. Copy the following script into the Dockerfile. This Dockerfile applies specifically to a Python app. If you are using another type of code, your Dockerfile must include a different base image and might require other fields to be defined.

Use the Python image from DockerHub as a base image
FROM python:3-slim

Expose the port for your python app
EXPOSE 5000

Copy all app files from the current directory into the app
directory in your container. Set the app directory
as the working directory
WORKDIR /cf-py/
COPY . .

Install any requirements that are defined
RUN pip install --no-cache-dir -r requirements.txt

Update the openssl package
RUN apt-get update && apt-get install -y 
openssl

Start the app.
CMD ["python", "welcome.py"]
: You can now download the custom connector SDK package from your Discovery for Cloud Pak for Data cluster, instead of retrieving the images and the SDK package from the Docker registry. For more information, see [Downloading the custom-crawler-docs.zip file in Discovery 2.2.1 and later](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-devdownload-ccs-zip).

Change to Invoices and Purchase orders
: Invoices and Purchase orders models can no longer be enabled in the tooling. If you need these models, please contact [IBM Cloud Support](https://cloud.ibm.com/unifiedsupport/supportcenter) to obtain instructions for enabling these models.

Change to Contracts enrichment tables
: In a Document Retrieval project that has the Contracts enrichment applied, tables are not included inside the contracts field, as they were previously in projects that had the Contracts enrichment enabled. Tables will continue to be included in a separate tables field when the Table Understanding enrichment is applied.

Change to support for Oracle Database 11g and Postgres 9.5
: Support for connecting to Oracle Database 11g was removed because the vendor ended version support on 31 December 2020.
: Support for connecting to Postgres 9.5 was removed because the vendor ended version support on 11 February 2021.





 2.2.0 release, 8 December 2020 

New release now available
: IBM Watson™ Discovery for IBM Cloud Pak for Data version 2.2 is available.
: Discovery for Cloud Pak for Data now works with IIBM Cloud Pak® for Data 3.5.

New support for Notes attachments
: Added support for attachments in the Notes data source. For more information, see [Notes](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconnectnotes)

New web crawl scheduling option
: You can specify the exact time that you would like your crawls to run for any data source, giving you the flexibility to run them at the times you prefer.
Objects for arguments 

The methods of your SDK that represent operations of your service should take a single argument - an instance of an "options" struct. Each operation should have its own options struct which contains fields that represent the operation parameters.





 Standard features 



 Authentication 

You are not required to use a particular library to provide the authentication for your service.

Your SDK must support all of the authentication methods supported by your service.





 Configuration 

Your SDK should allow various SDK configuration properties (e.g. service URL, authentication credentials) to be supplied through an external configuration mechanism (e.g. properties file or environment variables) in addition to a programmatic path. This will relieve SDK users from focusing on configuration-related functionality and will allow applications using your SDK to be flexible and avoid 'hard-coding' configuration information inside the application code.

If you build this capability into your SDK, you must document this mechanism clearly with examples.





 Using go-sdk-core 

[IBM go-sdk-core](https://github.com/IBM/go-sdk-core) provides configuration and authentication support. You can use the existing functionality provided by this dependency in your SDK.







 Documentation 

Your SDK is not useful if your audience cannot understand how to consume it in order to do the basic operations for your service. Your SDK needs to contain the following resources to help your users:



* README.md
* [Contributor guidelines](https://cloud.ibm.com/docs/sdk-handbook?topic=sdk-handbook-documentationsdk-contributor-docs)
* Code Samples
* [Service documentation](https://cloud.ibm.com/docs/sdk-handbook?topic=sdk-handbook-documentation)







 Samples 

For your Go SDK, you will want some simple code samples and explanations of what each does. Linking out to the API reference documentation for more advanced use is strongly encouraged.

At minimum, the samples should be included in the README.md file. They should communicate how to install the library, and complete the basic operations provided by your API.