vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Build a data lake using object storage 

This tutorial may incur costs. Use the [Cost Estimator](https://cloud.ibm.com/estimator/review) to generate a cost estimate based on your projected usage.

Definitions of the term data lake vary, but in the context of this tutorial, a data lake is an approach to storing data in its native format for organizational use. To that end, you will create a data lake for your organization using Object Storage. By combining Object Storage and Data Engine, data analysts can query data where it lies using Structured Query Language (SQL). You'll also leverage the Data Engine (previously SQL Query) service in a Jupyter Notebook to conduct a simple analysis. When you're done, allow non-technical users to discover their own insights.



 Objectives 



* Use Object Storage to store raw data files
* Query data directly from Object Storage using Data Engine (previously SQL Query)
* Refine and analyze data in IBM Watson® Studio



Zoom

![Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution29/Smart-Data-Lake-Architecture.png)

Figure 1. Architecture diagram of the tutorial



1. Raw data is stored on Object Storage.
2. Data is reduced, enhanced or refined with Data Engine.
3. Data analysis occurs in Watson Studio.
4. Non-technical users access data through application(s).
5. Refined data is pulled from Object Storage.







 Before you begin 

This tutorial requires:



* IBM Cloud CLI.



You will find instructions to download and install these tools for your operating environment in the [Getting started with tutorials](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-tutorials) guide.

To avoid the installation of these tools you can use the [Cloud Shell](https://cloud.ibm.com/shell) from the IBM Cloud console.
[solution icon](https://cloud.ibm.com/media/docs/images/homepage/magic-wand.svg) [Best practices for organizing users, teams, applications](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-users-teams-applications)Best practices for organizing users, teams, applications Solution tutorial

This tutorial gives an overview of the concepts available in IBM Cloud for identity and access management and how they can be implemented to support the multiple development stages of an application.

![solution tutorial icon](https://cloud.ibm.com/media/docs/images/homepage/solution-tutorial.svg) [Bring Your Own IP Address](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-byoip)Bring Your Own IP Address Solution tutorial

This tutorial presents a brief overview of BYOIP implementation patterns that can be used with IBM Cloud and a decision tree for identifying the appropriate pattern when realizing the secure enclosure as described in the Isolate workloads with a secure private network tutorial. Setup may require additional input from your onsite network team, IBM Cloud technical support or IBM Services.

![solution tutorial icon](https://cloud.ibm.com/media/docs/images/homepage/solution-tutorial.svg) [Build a data lake using object storage](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-smart-data-lake)Build a data lake using object storage Solution tutorial

Definitions of the term data lake vary, but in the context of this tutorial, a data lake is an approach to storing data in its native format for organizational use. To that end, you will create a data lake for your organization using Object Storage. By combining Object Storage and Data Engine, data analysts can query data where it lies using Structured Query Language (SQL). You'll also leverage the Data Engine (previously SQL Query) service in a Jupyter Notebook to conduct a simple analysis. When you're done, allow non-technical users to discover their own insights.

!
Congratulations, you have built a data lake using Object Storage. Below are additional suggestions to enhance your data lake.



* Experiment with additional datasets using Data Engine
* Stream data from multiple sources into your data lake by completing [Big data logs with streaming analytics and SQL](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-big-data-log-analyticsbig-data-log-analytics)
* Build a web app with a dashboard for line of business users utilizing [IBM Cognos Dashboard Embedded](https://cloud.ibm.com/docs/cognos-dashboard-embedded?topic=cognos-dashboard-embedded-gettingstartedtutorial).







 Step 8: Remove resources 

Run the following commands to remove services, applications and keys you created and used.

ibmcloud resource service-instance-delete data-lake-sql

ibmcloud resource service-instance-delete data-lake-studio

ibmcloud iam api-key-delete data-lake-cos-key

ibmcloud resource service-instance-delete data-lake-cos

If the deletion of data-lake-cos is not successful delete it from the storage section of the [Resource List](https://cloud.ibm.com/resources).

Depending on the resource it might not be deleted immediately, but retained (by default for 7 days). You can reclaim the resource by deleting it permanently or restore it within the retention period. See this document on how to [use resource reclamation](https://cloud.ibm.com/docs/account?topic=account-resource-reclamation).





 Related content 



* [ibmcloudsql](https://github.com/IBM-Cloud/sql-query-clients/tree/master/Python)
* [Jupyter Notebooks](https://jupyter.org/)
* [Folium](https://python-visualization.github.io/folium/)
HeatMap(locations, radius=15).add_to(m)
m

Zoom

![Notebook](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution29/notebook-mapbox.png)

Notebook
3. Click File > Save to save your Notebook to Object Storage.







 Step 6: Share your dataset with the organization 

Not every user of the data lake is a data scientist. You can allow non-technical users to gain insight from the data lake. Tools with analytic capabilities or for visualization can import data stored in CSV files. Application developers can make use of [IBM Cognos Dashboard Embedded](https://cloud.ibm.com/docs/cognos-dashboard-embedded?topic=cognos-dashboard-embedded-gettingstartedtutorial) to let users build and use feature-rich dashboards. Such a dashboard for the traffic data is shown below.

Zoom

![Dashboard Chart](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution29/dashboard-chart.png)

Dashboard Chart





 Step 7: Expand the tutorial 

Congratulations, you have built a data lake using Object Storage. Below are additional suggestions to enhance your data lake.



* Experiment with additional datasets using Data Engine
* Stream data from multiple sources into your data lake by completing [Big data logs with streaming analytics and SQL](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-big-data-log-analyticsbig-data-log-analytics)
Overview 

IBM Cloud® Data Engine is a fully managed service that runs SQL queries (that is, SELECT statements) to read, analyze, transform, store, and stream data in IBM Cloud® Object Storage and Kafka. You can also manage table metadata in a catalog that is compatible with Hive metastore. Data Engine is IBM Cloud®'s central service for data lakes. Combining Data Engine with data in Object Storage enables you to create an active workspace for a range of big data analytics use cases.

Zoom

![Data Engine overview.](https://cloud.ibm.com/docs-content/v1/content/9fbd9f26e0f1e4aaa38411d9a81ad7bc6897624d/sql-query/images/streams_landing_DE.svg)

Figure 1. Data Engine overview

Input data is read from CSV, JSON, ORC, Parquet, or AVRO objects located in one or more Cloud Object Storage instances. Each query result is written to a CSV, JSON, ORC, Parquet, or AVRO object in a Cloud Object Storage or Db2 instance of your choice. Use the Data Engine user interface (UI) to [develop your queries](https://cloud.ibm.com/docs/sql-query?topic=sql-query-running) and the [Data EngineREST API](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewrestapi) to automate them.



 Input and output of queries 

Before you can use the Data Engine service to run SQL queries, the input data must be uploaded to one or more Cloud Object Storage instances. You must also have at least 'Writer' access to at least one Cloud Object Storage bucket, so that result objects (that is, the objects that contain output data) can be written there.
Overview 

IBM Cloud® Data Engine is a fully managed service that runs SQL queries (that is, SELECT statements) to read, analyze, transform, store, and stream data in IBM Cloud® Object Storage and Kafka. You can also manage table metadata in a catalog that is compatible with Hive metastore. Data Engine is IBM Cloud®'s central service for data lakes. Combining Data Engine with data in Object Storage enables you to create an active workspace for a range of big data analytics use cases.

Zoom

![Data Engine overview.](https://cloud.ibm.com/docs-content/v1/content/9fbd9f26e0f1e4aaa38411d9a81ad7bc6897624d/sql-query/images/streams_landing_DE.svg)

Figure 1. Data Engine overview

Input data is read from CSV, JSON, ORC, Parquet, or AVRO objects located in one or more Cloud Object Storage instances. Each query result is written to a CSV, JSON, ORC, Parquet, or AVRO object in a Cloud Object Storage or Db2 instance of your choice. Use the Data Engine user interface (UI) to [develop your queries](https://cloud.ibm.com/docs/sql-query?topic=sql-query-running) and the [Data EngineREST API](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overviewrestapi) to automate them.



 Input and output of queries 

Before you can use the Data Engine service to run SQL queries, the input data must be uploaded to one or more Cloud Object Storage instances. You must also have at least 'Writer' access to at least one Cloud Object Storage bucket, so that result objects (that is, the objects that contain output data) can be written there.
A region that is spread across data centers in multiple zones to increase fault tolerance. See also [zone](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2070723).





 MZR 

See [multizone region](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx9774820).







 N 



 named entity 

A concept in a domain that falls in to a well defined category, such as names of organizations, locations, authors, or diseases.





 namespace 

A collection of repositories that store images in a registry. A namespace is associated with an IBM Cloud account, which can include multiple namespaces. See also [image](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2024928), [private image repository](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx8439215).





 NAT 

See [network address translation](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2031199).





 natural language processing (NLP) 

A field of artificial intelligence and linguistics that studies the problems inherent in the processing and manipulation of natural language, with an aim to increase the ability of computers to understand human languages.





 network address translation (NAT) 

An addressing method that is used to enable one IP address to communicate with several other IP addresses, such as those on a private subnet, by means of a lookup table. Network address translation has two main types: 1-to-1 and many-to-1.





 Network File System (NFS) 

A protocol that allows a computer to access files over a network as if they were on its local disks.





 NFS 

See [Network File System](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2031282).





 NLP
Using IBM Cloud Data Engine as external metastore 

IBM Cloud Data Engine is IBM Cloud's central service for data lakes. It provides stream ingestion, data preparation, ETL, and data query from IBM Cloud Object Storage and Kafka. It also manages tables and views in a catalog that is compatible with Hive metastore and other big data engines and services can connect to it. See [Overview of IBM Cloud Data Engine](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).

Each instance of IBM Cloud Data Engine includes a database catalog that you can use to register and manage table definitions for your data on IBM Cloud Object Storage. Catalog syntax is compatible with Hive metastore syntax. You can use IBM Cloud Data Engine to externalize metadata outside the IBM Analytics Engine Spark cluster.



 Pre-requisites 

The following are the pre-requisites:



* Creating IBM Cloud Data Engine instance
* Storing data in Cloud Object Storage
* Creating schema





 Creating IBM Cloud Data Engine instance 

{: metastore-prerequisite_line1}

Create an IBM Cloud Data Engine instance by using the Standard plan. See [Data Engine](http://cloud.ibm.com/catalog/services/data-engine-previously-sql-query).

After you have provisioned the Data Engine instance:
1. Make a note of the CRN of the instance.
1. Create an account-level API key or service ID level API key with access to the instance.
1. This service ID should be granted access to both the Data Engine instance as well as the IBM Cloud Object Storage bucket.

You can then configure your IBM Analytics Engine instance to use the default metastore configuration either at instance level or at application level as needed.

IBM Cloud Data Engine supports creating instances for different endpoints(location). Within an instance, different IBM Cloud Object Storage buckets are created to store data. The data buckets can be created for different end points(region). The endpoints for the data engine instance(thrift) and the data bucket are different.
Datastore clusters are to datastores what clusters are to hosts. When you create a datastore cluster, you can use vSphere Storage DRS to manage storage resources. When you add a datastore to a datastore cluster, the datastore's resources become part of the datastore cluster's resources. Use datastore clusters to aggregate storage resources, providing support resource allocation policies at the datastore cluster level. For more information, see [Creating a datastore cluster](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-598DF695-107E-406B-9C95-0AF961FC227A.html). 







 Network procedures 



Table 6. Network procedures

 Title Description 

 Network considerations For more information, see [Networking considerations for vCenter Server instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_networkingonvcenterserver). 
 Planning for HCX VMware Hybrid Cloud Services (HCX), allows disparate instances of vSphere software defined data centers (SDDC) to interoperate across various network types. HCX is designed to address the security, compatibility, complexity, and performance concerns one would encounter in trying to achieve a multi-instance, multisite, deployment of vSphere extending across on-premises and cloud provider boundaries. For more information, see [Preparing the installation environment](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-planning-prep-install). 
 Initial NSX configuration As part of the deployment of your vCenter Server instance, a sample customer network is available that consists of a private subnet, a public subnet, an NSX logical switch, a distributed logical router, and an NSX edge appliance that is deployed and configured to perform network address translation.
Table 3. Options for filtering by category

 Option Description 

 AI / Machine Learning Products that enable systems to learn from data rather than through explicit programming 
 Analytics Products that facilitate the analysis of data, typically large sets of business data, by the use of mathematics, statistics, and other means 
 Blockchain Products that facilitate the process of recording transactions and tracking assets in a business network 
 Compute Infrastructure resources that serve as the basis for building apps in the cloud 
 Containers A standard unit of software that packages up code and all its dependencies so the app runs quickly and reliably from one computing environment to another 
 Converged infrastructure A way of structuring an information technology (IT) system by grouping multiple components into a single optimized computing package, components of which may include servers; data storage devices; networking equipment; and software for IT infrastructure management, automation and orchestration 
 Databases Products that provide some form of access to a database without the need for setting up physical hardware, installing software, or configuring for performance 
 Developer tools Products that support developing, testing, and debugging software 
 Enterprise applications Bundle of compatible products that deliver enterprise-grade app solutions for information sharing, automation, and agility 
 Integration Products that facilitate the connection of data, apps, APIs, and devices across an organization to be more efficient, productive, and agile 
 Internet of Things Products that support receiving and transferring data over wireless networks without human intervention 
 Logging and monitoring Products that support storing, searching, analyzing, and monitoring log data and events. And, products that support reviewing and managing the operational workflow and processes being logged 
 Mobile Products with specific or special utility for users creatings things to be used on mobile devices 
 Networking Products that support or augment the linking of computers so they can operate interactively 
 Security Products that provide the protection of stored data from theft, leakage, and deletion 
 Storage Products that support data to be created, read, updated, and deleted 



You can also scope your view of the catalog by using the Provider filter to browse by individual providers and the Industry filter to view products catered for certain industries.







 Pricing and billing 

You can view the pricing details for each service when you're browsing the catalog. If you choose a service plan with a paid plan, you can estimate your costs by using the cost estimator tool.
Use the following table to identify which datasets are included in your plan and the range of historical data you can query.

<-- <table> -->

Datasets included in your plan

 Dataset Trial / Standard Enterprise / Security / GLB 

 firewallEventsAdaptiveGroups 30 days 30 days 
 firewallEventsAdaptive 30 days 30 days 

<-- </table ""> -->

--filter: Filter events. The default value is the last 6 hours of data.The following operators are supported for all filter options:<-- <table> -->Operators supported for filter options| Operator | Comparison || -------- | ------------------- || gt | greater than || lt | less than || geq | greater or equal to || leq | less or equal to || neq | not equal || in | in |<-- </table ""> --> - firewallEventsAdaptiveGroups filter options.
- datetime
- datetimeFifteenMinutes
- datetimeHour
- datetimeFiveMinutes
- datetimeMinute
- matchIndex
- sampleInterval
- The following filter options support like and notlike operators.
- action
- clientASNDescription
- clientAsn
- clientCountryName
- clientIP
- clientRefererHost
- clientRefererPath
- clientRefererQuery
- clientRefererScheme
- clientRequestHTTPHost
- clientRequestHTTPMethodName
- clientRequestHTTPProtocol
- clientRequestPath
- clientRequestQuery
- clientRequestScheme
- edgeColoName
- edgeResponseStatus
- kind
- originResponseStatus
- originatorRayName
- rayName
- ref
- ruleId
- source
- userAgent
Show more
--order: Output order.
Geography Data center Pod Server options<br><br>[1] 

 Asia-Pacific CHE01 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA21 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA22 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA23 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SNG01 02 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SYD01 01-02 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SYD04 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SYD05 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific TOK02 01-02 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific TOK04 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific TOK05 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 







 Related links 



* [Ordering VMware vSphere instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_orderinginstances-req)
* [Adding ESXi servers to VMware vSphere instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_addingservers)
Data centers host the power, cooling, compute, network, and storage resources used for services and apps. They don't provide isolation from multizones in a location.

Data centers are based on a POD architecture where each data center can have more than one POD, depending on the on-demand build out. Each POD consists of racks, servers, networks, and storage, along with backup power generators. Placing application servers across PODs improves the availability.

Zoom

![Global map showing MZR and data center locations](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/overview/images/Global-View.svg)

Figure 3. MZR and data center locations map

This image is an artistic representation and does not reflect actual political or geographic boundaries.

See the following table for the specific code for each data center.

Americas

Europe

Asia Pacific



Table 3. Data centers in North and South America
Use the buttons before the table to change the context of the table. The column headers identify the data centers located in the specific geographical area.

 Data center Code 

 Dallas 05 DAL05 
 Dallas 08 <br><br>[1] DAL08 
 Dallas 09 DAL09 
 Dallas 10 DAL10 
 Dallas 12 DAL12 
 Dallas 13 DAL13 
 Montreal 01 MON01 
 San Jose 01 SJC01 
 San Jose 03 SJC03 
 San Jose 04 SJC04 
 Sao Paulo 01 SAO01 
 Sao Paulo 04 SAO04 
 Sao Paulo 05 SAO05 
 Toronto 01 TOR01 
 Toronto 04 TOR04 
 Toronto 05 TOR05 
 Washington DC 01 WDC01 
 Washington DC 03 <br><br>[2] WDC03 
 Washington DC 04 WDC04 
 Washington DC 06 WDC06 
 Washington DC 07 WDC07 



The table includes certain data centers that are set to close soon.
* name: the DNS hostname to associate with your Load Balancer.
* fallback_pool: the pool ID to use when all other pools are detected as unhealthy.
* default_pools: a list of pool IDs ordered by their failover priority.

<-- </ul> -->

* The optional fields are description, ttl, region_pools, proxied, enabled, session_affinity, session_affinity_ttl, steering_policy:

<-- <ul> -->

* description: the description of your Load Balancer.
* ttl: time to live (TTL) of the DNS entry for the IP address returned by this load balancer.
* region_pools: a mapping of region/country codes to a list of pool IDs (ordered by their failover priority) for the given region.
* proxied: Control whether or not traffic should flow through the security and performance functions on CIS.
* enabled: Whether to enable (the default) this load balancer.
* session_affinity: valid values are cookie, none.
* session_affinity_ttl: Time, in seconds, until this load balancers session affinity cookie expires after being created.Valid value is between [1800, 604800\. Default is 82800.
* steering_policy: valid values for steering_policy are off, geo, random, dynamic_latency.

<-- <ul> -->

* off: use default_pools.
* geo: use region_pools/pop_pools.
* random: select a pool randomly.
* dynamic_latency: use round trip time to select the closest pool in default_pools (requires pool health checks).

<-- </ul> -->

<-- </ul> -->

<-- </ul> -->

Sample JSON data: {
"name": "www.example.com",
"fallback_pool": "17b5962d775c646f3f9725cbc7a53df4",
"default_pools":