vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you specify https://www.example.com/banking/faqs.html, all URLs that begin with https://www.example.com/banking/ are crawled, for example. If you specify https://www.example.com/banking all URLs that begin with https://www.example.com/ are crawled.

By default, the number of consecutive links that the crawl follows from the starting URL is 2. To change the number of hops or to list website sections to exclude from the crawl, click the edit icon.



* The maximum number of hops allowed is 20.
* To specify URL paths to exclude, add the site path. For example, if the starting URL is https://example.com, you can exclude https://example.com/pricing by entering /pricing/.

Any section of the web address that contains the site path you specify is excluded. For example, if you specify /licenses/, the page https://example.com/products/licenses/europe is excluded, among others.
* If you want to restrict the crawl to a single page, add the URL to the Starting URLs field. For example, https://www.example.com/banking/faqs.html. Click the edit icon to set the Maximum number of links to follow to 0.



* If the website that you want to crawl uses JavaScript to customize the page content before it is displayed, you must take an extra step.

After you enter the starting URL and click Add, edit the URL by clicking the edit icon ![Edit icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/edit.svg). Set the Execute JavaScript during crawl switcher to On, and then click Save.

When JavaScript processing is enabled, it takes 3 to 4 times longer to crawl a page. Use it only on individual web pages where you know it is necessary because the page renders its content dynamically.
When enabled, everything stops until the processing step is completed successfully.



* If you have a critical postprocessing step that must be taken before you want to allow the response to be sent to the customer, then enable this setting.
* When this setting is disabled, the assistant ignores any errors it encounters and continues to process the response without taking the processing step. If the postprocessing step is helpful but not critical, consider keeping this setting disabled.



6. In the URL field, add the URL for the external application to which you want to send HTTP POST request callouts.

For example, maybe you store your assistant's responses in a separate content management system. When the assistant understands the input, the processed action returns a unique ID that corresponds to a response in your CMS. To call a service that retrieves a response from your CMS for a given unique ID, specify the URL for your service instance (for example, https://example.com/get_answer.

You must specify a URL that uses the SSL protocol, so specify a URL that begins with https.

You cannot use a webhook to call a Cloud Functions action that uses token-based Identity and Access Management (IAM) authentication. However, you can make a call to a Cloud Functions web action or a secured web action.
7. In the Secret field, add a private key to pass with the request that can be used to authenticate with the external service.

The key must be specified as a text string, such as purple unicorn. The maximum length is 1,024 characters. You cannot specify a context variable.

It is the responsibility of the external service to check for and verify the secret. If the external service does not require a token, specify any string you want. You cannot leave this field empty.

If you want to see the secret as you enter it, click on the Show password icon ![view icon](https://cloud.ibm.com/docs-content/v1/content/icons/view.svg) before you start typing. After you save the secret, the string is replaced by asterisks and can't be viewed again.
8.
Look especially at the Containers section.
* Vorbis (audio/webm;codecs=vorbis). For more information, see [xiph.org/vorbis](https://xiph.org/vorbis/) and [Vorbis](https://wikipedia.org/wiki/Vorbis).



If you omit the codec, the service automatically detects it from the input audio.

For JavaScript code that shows how to capture audio from a microphone in a Chrome browser and encode it into a WebM data stream, see [jsbin.com/hedujihuqo/edit?js,console](https://jsbin.com/hedujihuqo/edit?js,console). The code does not submit the captured audio to the service.







 Specifying an audio format 

You use the Content-Type request header or equivalent parameter to specify the format (MIME type) of the audio that you send to the service. You can specify the audio format for any request, but that's not always necessary:



* For most formats, the content type is optional. You can omit the content type or specify application/octet-stream to have the service automatically detect the format.
* For other formats, the content type is required. These formats do not provide the information, such as the sampling rate, that the service needs to auto-detect their format. You must specify a content type for the audio/alaw, audio/basic, audio/l16, and audio/mulaw formats.



For more information about the formats that require a content-type specification, see Table 1 in [Audio formats](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-list). The table's Content-type specification column indicates whether you must specify the content type.

For examples of specifying a content type with each of the service's interfaces, see [Making a speech recognition request](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-request).
Install [Mozilla Firefox](https://www.mozilla.org/), [Google Chrome](https://www.google.com/intl/us_en/chrome/) or [Microsoft Edge](https://www.microsoft.com/en-us/edge) into your Jump server. One of these browsers is required e.g. to access hosts or vCenter later in this tutorial.

You may need to use SSH later when configuring, managing or configuring various VMware assets. SSH is not required in this tutorial, but it is useful. You may use your favorite SSH client in the Jump server, such as [PuTTY](https://www.putty.org) or [mRemoteNG](https://mremoteng.org).







 Next steps 

The next step in the tutorial series is:



* [Provision IBM Cloud DNS Services for VMware deployment](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-bm-vmware-dnsvpc-bm-vmware-dns)
// Hosts don't match. This is a hotlink. Redirect the
// user to our homepage.
return new Response('', {
status: 302,
headers: {
'Location': '/'
}
})
}
}

// Everything is fine, return the response normally.
return response
}
Show more





 Originless responses 

You can return responses directly from the edge. No need to hit your origin.



 Ignore POST and PUT HTTP requests 

Ignore POST and PUT HTTP requests. This snippet allows all other requests to pass through to the origin.

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

async function fetchAndApply(request) {
if (request.method === 'POST' || request.method === 'PUT') {
return new Response('Sorry, this page is not available.',
{ status: 403, statusText: 'Forbidden' })
}

return fetch(request)
}





 Deny a spider or crawler 

Protect your origin from unwanted spiders or crawlers. In this case, if the user-agent is “annoying-robot”, the Edge function returns the response instead of sending the request to the origin.

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

async function fetchAndApply(request) {
if (request.headers.get('user-agent').includes('annoying_robot')) {
return new Response('Sorry, this page is not available.',
{ status: 403, statusText: 'Forbidden' })
}

return fetch(request)
}





 Prevent a specific IP from connecting 

Blocklist IP addresses. This snippet of code prevents a specific IP, in this case ‘225.0.0.1’ from connecting to the origin.
When enabled, everything stops until the processing step is completed successfully.



* If you have a critical postprocessing step that must be taken before you want to allow the response to be sent to the customer, then enable this setting.
* When this setting is disabled, the assistant ignores any errors it encounters and continues to process the response without taking the processing step. If the postprocessing step is helpful but not critical, consider keeping this setting disabled.



5. In the URL field, add the URL for the external application to which you want to send HTTP POST request callouts.

For example, maybe you store your assistant's responses in a separate content management system. When the skill understands the input, the processed dialog node returns a unique ID that corresponds to a response in your CMS. To call a service that retrieves a response from your CMS for a given unique ID, specify the URL for your service instance.

https://my-webhook.com/get_answer

You must specify a URL that uses the SSL protocol, so specify a URL that begins with https.

You cannot use a webhook to call a Cloud Functions action that uses token-based Identity and Access Management (IAM) authentication. However, you can make a call to a Cloud Functions web action or a secured web action.
6. In the Secret field, add a private key to pass with the request that can be used to authenticate with the external service.

The key must be specified as a text string, such as purple unicorn. Maximum length is 1,024 characters. You cannot specify a context variable.

It is the responsibility of the external service to check for and verify the secret. If the external service does not require a token, specify any string you want. You cannot leave this field empty.

If you want to see the secret as you enter it, click on the Show password icon ![view icon](https://cloud.ibm.com/docs-content/v1/content/icons/view.svg) before you start typing. After you save the secret, the string is replaced by asterisks and can't be viewed again.
7.
"ClientRequestUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0",
"ClientSSLCipher":"NONE",
"ClientSSLProtocol":"none",
"ClientSrcPort":48718,
"EdgeColoID":134,
"EdgeEndTimestamp":"2019-01-04T02:22:26Z",
"EdgePathingOp":"wl",
"EdgePathingSrc":"macro",
"EdgePathingStatus":"nr",
"EdgeRateLimitAction":"",
"EdgeRateLimitID":0,
"EdgeRequestHost":"",
"EdgeResponseBytes":1849,
"EdgeResponseCompressionRatio":2.82,
"EdgeResponseContentType":"text/html",
"EdgeResponseStatus":403,
"EdgeServerIP":"",
"EdgeStartTimestamp":"2019-01-04T02:22:26Z",
"OriginIP":"",
"OriginResponseBytes":0,
"OriginResponseHTTPExpires":"",
"OriginResponseHTTPLastModified":"",
"OriginResponseStatus":0,
"OriginResponseTime":0,
"OriginSSLProtocol":"unknown",
"ParentRayID":"00",
"RayID":"493a3cc9463eb0d4",
"SecurityLevel":"med",
"WAFAction":"drop",
"WAFFlags":"0",
"WAFMatchedVar":"ARGS:USERNAME",
"WAFProfile":"off",
const ua = request.headers.get('user-agent')
if (ua.match(/iphone/i) || ua.match(/ipod/i)) {
uaSuffix = '/mobile'
} else if (ua.match(/ipad/i)) {
uaSuffix = '/tablet'
}

return fetch(request.url + uaSuffix, request)
}
Show more







 Hot-link protection 

You can use CIS Edge functions to protect your hot-links on your web properties.

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

/
* If the browser is requesting an image and
* the referer does not match your host
* we redirect the request to your page
/
async function fetchAndApply(request) {
// Fetch the response.
let response = await fetch(request)

// If it's an image, engage hotlink protection based on the
// Referer header.
let referer = request.headers.get('Referer')
let contentType = response.headers.get('Content-Type') || ''
if (referer && contentType.startsWith('image/')) {
// It's an image and there's a Referer. Verify that the
// hostnames match.
if (new URL(referer).hostname !==
new URL(request.url).hostname) {
// Hosts don't match. This is a hotlink. Redirect the
// user to our homepage.
return new Response('', {
status: 302,
headers: {
'Location': '/'
}
})
}
}

// Everything is fine, return the response normally.
return response
}
Show more





 Originless responses
Steps in a journey are shown to the customer one at a time, in the order in which you list them in the steps array.

As with assistant responses, the response_type property identifies the type of response:

text
: A step that shows only text.

{
"response_type": "text",
"text": "This is the text of the response."
}

Markdown formatting and links are supported in text steps. For more information, see [Markdown formatting](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-web-chat-architectureweb-chat-architecture-markdown).

Note that the structure of a text step in a journey is different from the text response type for assistant responses. Instead of an array of text values, only a single text component is supported.

image
: A step that shows an image, along with an optional description.

{
"response_type": "image",
"source": "https://example.com/image.png",
"description": "This is the description of the image."
}

The source property must be the https: URL of a publicly accessible image. The specified image must be in .jpg, .gif, or .png format.

video
: A step that shows a video, along with an optional description.

{
"response_type": "video",
"source": "https://example.com/videos/example-video.mp4",
"description": "This is the description of the video."
}

The URL specified by the source property can be either of the following:



* The URL of a video file in a standard format such as MPEG or AVI. In the web chat, the linked video will render as an embedded video player.

HLS (.m3u8) and DASH (MPD) streaming videos are not supported.
* The URL of a video hosted on a supported video hosting service.
Using services in another region 

If you have a service instance that is created and bound to apps in one region, you can use this service instance in another region by one of the following methods:



*  Use the service credentials to configure your app instance directly. See [Connecting services to external apps](https://cloud.ibm.com/docs/account?topic=account-externalapp) for details.
*  Create a user-provided service as a bridge.



To use a service instance that exists in another region, complete the following steps:



1.  To switch to the region where the service instance exists, click the Navigation menu icon ![Navigation Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d8c6eec891cff72b666dc24bd3211810c77fb42/icons/icon_hamburger.svg) > Resource list. Then, expand the LOCATION menu and select the region.
2.  Retrieve the credentials and the connection parameters from the VCAP_SERVICES environment variable of the service instance in the region where the service exists. Complete the following steps:



1.  In the IBM Cloud Dashboard, click View all on your apps tile. The Overview page is displayed.
2.  In the navigation pane, click Credentials. The VCAP_SERVICES environment variable details are displayed. Record the JSON content for the service instance.



3.  Switch to the region where you want to use the service instance. Click the Navigation menu icon ![Navigation Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d8c6eec891cff72b666dc24bd3211810c77fb42/icons/icon_hamburger.svg) > Resource list. Then, expand the LOCATION menu and select the region where you want to use the service instance.
4.  Create a user-provided service instance by using the credentials and connection parameters that you recorded from the VCAP_SERVICES environment variable.
5.  Bind the user-provided service instance to your app by using the following command:

ibmcloud service bind myapp user-provided_service_instance
* regionCode: If known, the [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2) code for the first level region associated with the IP address of the incoming request. If it is not known, it is an empty string (for example, "TX").
* timezone: Timezone of the incoming request (for example, "America/Chicago").



All plans can set these features on outbound requests.



* cacheEverything: This option forces CIS to cache the response for this request, regardless of what headers are seen on the response. This is equivalent to setting the page rule "Cache Level" to "Cache Everything" (for example, true).
* scrapeShield: Toggles ScrapeShield (for example, false).
* polish: Set Cloudflare Polish mode. The possible values are "lossy", "lossless" or "off" (for example, lossless).
* minify: Website optimization to Enable/disable Cloudflare Autominify for various file types. The value is an object containing Boolean fields for javascript, css, and html (for example, { javascript: true, css: true, html: false }).
* mirage: Image optimization to enable/disable Cloudflare mirage. When you specify this option, the value should always be false (for example, false).
* cacheTtl: This option forces CIS to cache the response for this request, regardless of what headers are seen on the response. This is equivalent to setting two page rules: "Edge Cache TTL" and "Cache Level" (to "Cache Everything"; for example,300).
* resolveOverride: Redirects the request to an alternate origin server. You can use this to implement load balancing across several origins (for example,us-east.example.com). For security reasons, the hostname set in resolveOverride must be proxied on the same CIS zone of the incoming request. Otherwise, the setting is ignored.
if (request.headers.get('user-agent').includes('annoying_robot')) {
return new Response('Sorry, this page is not available.',
{ status: 403, statusText: 'Forbidden' })
}

return fetch(request)
}





 Prevent a specific IP from connecting 

Blocklist IP addresses. This snippet of code prevents a specific IP, in this case ‘225.0.0.1’ from connecting to the origin.

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

async function fetchAndApply(request) {
if (request.headers.get('cf-connecting-ip') === '225.0.0.1') {
return new Response('Sorry, this page is not available.',
{ status: 403, statusText: 'Forbidden' })
}

return fetch(request)
}







 Post requests 

Reading content from an HTTP POST request:

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

/
* Making a curl request that looks like
* curl -X POST --data 'key=world' example.com
* or
* curl -X POST --form 'key=world' example.com
/
async function fetchAndApply(request) {
try {
const postData = await request.formData();
return new Response(hello ${postData.get('key')})
} catch (err) {
return new Response('could not unbundle post data')
}
}
Show more

Creating an HTTP POST request from an Edge function:

addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})

/
EspañolFrançaisItaliano日本語한국어Português/Brasil简体中文繁體中文





 HTTP concepts * Last updated 2020-12-09 

HTTP/2 and HTTP/3 accelerate page load and are included in all IBM Cloud® Internet Services plans. HTTP/2 is enabled by default and requires an SSL certificate at the CIS edge network. Domains on Trial plans cannot disable HTTP/2.

A browser and web server automatically negotiate the highest protocol available, so HTTP/3 takes precedence over HTTP/2. CIS only uses HTTP/1.x between the origin web server and CIS.

To determine the protocol used for your connection, enter example.com/cdn-cgi/trace from a web browser or client and replace example.com with your domain name. If http=h2 appears in the results that are returned, the connection occurred over HTTP/2. Other possible values are http=http2+quic/99 for HTTP/3, and http=http/1.x for HTTP/1.x.



 HTTP/2 

HTTP/2 improves page load times through:



* Connection multiplexing - Retrieves multiple resources in a single network request. Responses are sent when resources are available to avoid slowing page rendering.
* HTTP header compression - Compresses headers and simplifies HTTP requests to avoid resending headers.
* HTTP/2 Server Push - To improve page load speed, CIS provides additional resources for a client to cache without waiting for additional requests.



Not all browsers support HTTP/2 and use HTTP 1.x instead. Connection multiplexing is on a per-domain basis.





 HTTP/3 

HTTP/3 enables fast, reliable, and secure connections. HTTP/3 encrypts internet transport by default using a protocol from Google called QUIC. Enable HTTP/3 via the Cloudflare Network app. Use the following methods to experiment with HTTP/3.



 Using Google Chrome as your HTTP/3 client 

To use Chrome to connect to your website over HTTP/3, first download and install the [latest Canary build](https://www.google.com/chrome/canary/).
Available actions are: disable_security, always_use_https, ssl, browser_cache_ttl, security_level, cache_level, edge_cache_ttl, bypass_cache_on_cookie, browser_check, server_side_exclude, email_obfuscation, automatic_https_rewrites, opportunistic_encryption, ip_geolocation, explicit_cache_control, cache_deception_armor, waf, forwarding_url, image_load_optimization, image_size_optimization, script_load_optimization, host_header_override, resolve_override. And some actions are limited to Enterprise plan: cache_on_cookie, disable_apps, disable_performance, minify, origin_error_page_pass_thru, response_buffering, true_client_ip_header, sort_query_string_for_cache, respect_strong_etag.

<-- </ul> -->

* The optional fields are priority, status:

<-- <ul> -->

* priority:A number that indicates the preference for a page rule over another. Default is 1.
* status:Status of the page rule. The valid values are active and disabled, default is disabled.

<-- </ul> -->

<-- </ul> -->

Sample JSON data: {
"targets":
{
"target": "url",
"constraint": {
"operator": "matches",
"value": "example.com/images/"
}
}
],
"actions":
{
"id": "ssl",
"value": "flexible"
},
{
"id": "browser_cache_ttl",
"value": 14400
},
{
"id": "security_level",
"value": "medium"
},
{
"id": "cache_level",
"value": "basic"
},
{
Browser requirements 

The following list specifies the minimum required browser software for IBM Watson® Knowledge Studio.



*  Google Chrome 83 and future stable fix packs (Chrome provides the best performance when using the ground truth editor )
*  Mozilla Firefox Extended Support Release (ESR) 68 and future fix packs

If you open Knowledge Studio in a Private Browsing window, disable the tracking protection feature or allow pop-ups for Knowledge Studio sites to ensure that all the application functions work properly.



For the best performance, use a screen resolution of at least 1024x1280.
How do I use the ICOS default index page with CDN? 

The CDN does not support the default index page for the ICOS objects because the ICOS S3 endpoint does not have the default index. You must specify the complete request path in the browser's address bar (for example, www.example.com/index.html). If you want the CDN to automatically access the default index page of the ICOS, create a CDN with Server type of origin instead of an Object Storage type. For example, if you have a ICOS with static website hosting endpoint xyz-bucket-name.s3-web.us-south.cloud-object-storage.appdomain.cloud, you can create a CDN www.example.com with server type of origin xyz-bucket-name.s3-web.us-south.cloud-object-storage.appdomain.cloud in path /, when a user opens the CDN URL www.example.com, the CDN edge server retrieves the content from xyz-bucket-name.s3-web.us-south.cloud-object-storage.appdomain.cloud/, and ICOS will return your default index page.