Functionbeat for Streaming Cloudwatch Events to ElasticSearch
Tags: functionbeat, logs, aws, elastic
Published: 2020-05-11 02:45:03

![hover](https://dev-to-uploads.s3.amazonaws.com/i/sb85wswksvoft5e3j90t.jpg)
<sub> Photo by [Bjørn Christian Tørrissen](https://commons.wikimedia.org/wiki/User:Uspn) </sub>

With various serverless and opensource options available, it's easier to spin up a concept into some kind of working model with comparatively quicker than earlier times. With this advancement it becomes essential to be able to effectively monitoring the different components and solution.
What makes monitoring easier than having single logcentral to be able to get a view of required metrics.

# Functionbeat for Monitoring
Yes, as an engineer, I would like to setup a dashboard with all required information. And functionbeat helps in achieving the same with as simple as writing a lambda to collect the logs from different sources. Functionbeat is one of Elastic's beat family allowing you to be able to stream logs from Kinesis, SQS, Cloudwatch (as of today) to single logcentral.

![Functionbeat Architecture](https://dev-to-uploads.s3.amazonaws.com/i/wf8zn52hyfaizez2kwao.png)

# Collect the Cloudwatch Logs
What we are focusing here is, functionbeat to read each row of cloudwatch logs and stream it to elasticsearch. With functionbeat deployed as serverless lambda to AWS, you should be able to achieve above.

![Functionbeat Cloudwatch logs](https://dev-to-uploads.s3.amazonaws.com/i/0juj04hva2fm1djux525.png)

In this post, we focus on collecting logs from cloudwatch group and shipping it to elasticsearch.
`lambda -> Cloudwatch Logs -> Functionbeat -> Elasticsearch`

##### Setup a lambda with cloudwatch monitoring
A lambda function with any business function generating logs and metrics to cloudwatch. 

##### Setup Functionbeat lambda
1. Download functionbeat distribution
`https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-installation.html`

2. Before deploying cloudwatch, ensure you have AWS profile set-up. Verify that the aws account and region are set correctly using
`aws configure`

3. Create S3 bucket from AWS console for the functionbeat to be uploaded to. The reference to created S3 bucket is provided as

```
  functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
  functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy"
```

   4\. For the functionbeat to be deployed, lambda should have required permissions to the AWS resources. Create IAM role with access permissions to
`ec2, es, s3`
For detailed IAM cloudformation code refer to [sample] (https://github.com/mnlth1/functionbeat/blob/master/sample/functionbeat-iam.yml)

   5\. Configure functionbeat in `functionbeat.yaml` and setup lambda to add triggers to cloudwatch group defined in #1. `functionbeat-cloudwatch` is the name of lambda function and `type` representing the trigger type with actual values listed under `triggers` configuration. We could configure multiple cloudwatch log group triggers as comma separated values.
```
  functionbeat.provider.aws.functions:
    - name: fb-cloudwatch
      enabled: true
      type: cloudwatch_logs
      
      role: arn:aws:iam::xxawsaccountidxx:role/lambda.functionbeat.role

      triggers:
      - log_group_name: /aws/lambda/sample-lambda-events
```
If you would want to deploy the lambda as part of private cloud set-up, look at configuring `virtual_private_cloud` with subnet details.

Now, lets look at configure the output of these logs to be shipped to. It only requires to configure the elasticsearch host and port to be configured. Provided the AWS connectivity to elasticsearch is already established, hook into the host. Else, use the local elasticsearch distribution to verify.
```
  output.elasticsearch:
    hosts: ["localhost:9200"]
    protocol: http
```
If the elasticsearch setup requires credentials, the same can be configured under `output.elasticsearch` with `username` and `password`. To verify the setup so far, we could simply run the command

<b>`./functionbeat test config -e` </b>

6. Finally, we are ready to deploy functionbeat to AWS. Deploy command uploads the functionbeat to s3 bucket `functionbeat-deploy` created as part of #3. Use the command:

<b>`./functionbeat -v -e -d "*" deploy fb-cloudwatch`</b> 

This would create cloudformation stack in AWS creating required lambda and resources. Once successfully deployed, you should be able to navigate to cloudwatch and look at the functionbeat `fb-cloudwatch` logs. You could enable debug logs on functionbeat to be able to view detailed logs, add config `logging.level: debug` to `functionbeat.yaml`. 

With any additional changes, you can update the functionbeat using `update` command without requiring to recreate the whole function.
<b>`./functionbeat -v -e -d "*" update fb-cloudwatch`</b>

##### Improvements
As part of the lambda function collecting and streaming logs, we could pick the required fields from log source. This could be configured as `processors` to the function. Say, if its required  to drop field `noMuchUse` from log group and enrich with host metadata, set up processor
```
  processors:
    - include_fields:
      fields: ["id"]
    - drop_fields: 
      fields: ["id.noMuchUse"]
    - add_host_metadata:
      netinfo.enabled: false
```

References: 
[Functionbeat](https://www.elastic.co/guide/en/beats/functionbeat/current/index.html)
[Github](https://github.com/mnlth1/functionbeat)