Amazon CloudWatch multi data source querying
Tags: monitoring, cloud, aws, cloudcomputing
Published: 2024-02-01 18:43:26

Amazon CloudWatch multi data source querying
Embracing a commitment to innovation and user empowerment, AWS CloudWatch continues to evolve with cutting-edge features that redefine the landscape of cloud monitoring and management. In this blog post, we delve into the latest advancements within AWS CloudWatch, showcasing how these enhancements empower users to gain deeper insights, optimize performance, and seamlessly navigate the dynamic world of cloud computing. 

Introduction

Gain visibility across your hybrid and multicloud metrics in a single view with Amazon CloudWatch multi data source querying. With this feature, you can consolidate and visualize metrics from sources such as Amazon OpenSearch Service, Amazon Managed Service for Prometheus, Azure Monitor, your own custom data sources, and query those metrics in real time, increasing visibility into your application health and helping you resolve critical events faster. 
As we know now CloudWatch can support multi data source querying. The source can be any of the options listed below.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lxcqyx0u27ui5h2r64kc.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kxkedu12g8nairecpdml.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8hoxm444z0xp3cqkak35.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s5s120os00qtnigqf9m1.jpg)

You have the flexibility to query, visualise, and set alarms for all metrics, irrespective of their origin.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zsyoi0d0d3pgv7io1tnc.jpg)
In this blog, we will explore the process of obtaining the source data from an S3 bucket and creating a dashboard—a practical use case that highlights the integration and visualization capabilities within the context of the discussed topic.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lz7k5kdy9cfhwwimfa0g.jpg)

## Step 1 Create bucket

Create bucket name cw-bucket1

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4kyciyt38tnr3uchcxb.jpg)

Upload the csv formatted file with the first column timestamp example below.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qt7r7ieniftss2sr4url.jpg)

Select the S3 bucket that you have created

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xxwk5i3ndb04t1ppp1hc.jpg)

In this example the file name is final.csv

Upload the final.csv file to this bucket

Finally it should look likes as below


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ppvwo57ti37fsugqkqma.jpg)

So all good to go for CloudWatch now

## Step 2 Cloudwatch multi source query initial setup

Access the CloudWatch service console by opening it. Navigate to the Metrics section and expand it. 

Then, proceed to click on "All Metrics."
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bns2fz81uqj3b1uaidf1.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7j73j2wo58pgofng38pm.jpg)
        
You will observe a display resembling the following.
Select Multi source query

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4vdmxu3kp03le9vu6e43.jpg)

Click “Cloudwatch Metrics Insights” pull dropdown menu

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aojjn51tn3kon6zjh8r0.jpg)

Select “Create and manage data sources”

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ubaord4rgw5ec2sbrxew.jpg)


It will look like the screenshot below. You can see the supported data sources.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jlut4ezt77pk2aoban0g.jpg)

Click on getting started.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t7csgip63e18kv36os6q.jpg)

Select S3 as the data source for the example and click next.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/271mt191rxopjnd76vxv.jpg)

Enter data source name project-csv

Acknowledged to the IAM resources which automatically created with CloudFormation.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w29of02i2rinkoe70y21.jpg)

Click  “Create data source” right side lower cornet.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9pt1k4ljr27ru523p5cl.jpg)

Once you click it will run a cloudformation template to create resources.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nw6kwn3nbwsrqsc3htqr.jpg)

Wait until the cloudformation template create the necessary resources.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0dtwf6lcbrn4a6b37amn.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7nmallar3germ59hgdst.jpg)


After completing the CloudFormation template the below screen will be displayed.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6xfteni2dbwekckd51p7.jpg)

Then the last step to point out the bucket and the key file.


## Step 3 Point out the source as bucket and key file.

Click on “Query from Cloudwatch Metrics” link

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bk7aqrv4d30xwljmkmej.jpg)
We need to select the bucket and key

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wh6iu5dxgvd50pj4gfca.jpg)

After adding the bucket name and key, it will look like below.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ums1vt6rva0qobd4focy.jpg)

The final steps to query

## Step 4 Cloudwatch multi source Query

Click on Graph Query

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/izvka7z2f7ttyz31lna0.jpg)

In case required adjust the time range from top

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3tlpvn16qezpttg56zr7.jpg)

Based on your data the graph will be displayed

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eu9e949rsr3n0n0iol9w.jpg)

You can play with different kinds of graphs which suit the data.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k26qqh4d4s211cp850z7.jpg)

So finally we finished the Cloudwatch group query with source as S3 Bucket.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d7zy4aqyp74s1zdmuunw.jpg)

## Step 5  Final steps is clean up

Open CloudFormation console.

Note : Do not forget to check the Lambda function dashboard.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ww4zp8fru61y54aox2lc.jpg)

Just delete the CloudFormation template. Wait till the Cloudformation delete is completed.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mbvbxuwo1az0b53zz8f.jpg)

Open the S3 console.
Empty the bucket cw-bucket1 and delete the bucket.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l3fktqtcced236bz5t55.jpg)

Lambda function screenshots Additional.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gd4c1laozxgpcwek2nks.jpg)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/la512l47udituo4y8fye.jpg)

Conclusion 

In conclusion, CloudWatch enables seamless querying of S3 functions without the need for additional code. This blog has provided insights into one of the sources for CloudWatch's multi-source query capabilities, focusing specifically on S3 integration. Stay tuned for future blog posts, where we will explore and delve into other data sources, expanding our understanding of CloudWatch's versatile querying capabilities. The overall takeaway emphasises the simplicity and efficiency with which CloudWatch handles multi-source queries, opening avenues for comprehensive monitoring and analysis across various data streams.


Reference
Read the AWS News Blog and watch one more demo for multi source in the What’s new with AWS observability and operations breakout.