AWS: RDS logs, export to the CloudWatch Logs and CloudFormation template
Tags: aws, amazonwebservices, cloudformation, cloudwatch
Published: 2019-03-07 17:48:57

---
title: AWS: RDS logs, export to the CloudWatch Logs and CloudFormation template
published: true
tags: AWS,Amazon Web Services,CloudFormation,CloudWatch
canonical_url: https://rtfm.co.ua/en/aws-rds-logs-export-to-the-cloudwatch-logs-and-cloudformation-template/
---

![](https://rtfm.co.ua/wp-content/uploads/2014/11/aws-logo-square-02-e1417012834176.png)We have a bunch of the AWS RDS with MariaDB.

Backend-developers asked me to enable Slow requests logs so they can debug their application.

So the task is: enable AWS RDS logging and configure export to the CloudWatch Logs for further analysis.

As everything else – our RDS instances are configured via CloudFormation templates, so will add some examples as well.

Beside standard general/error/slow logs – there is also Audit Plugin for MariaDB to enable Audit-log, but this is topic for another post. You can check its documentation [here>>>](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MariaDB.Options.html#Appendix.MariaDB.Options.AuditPlugin.Options).

### DB Parameter Groups – enabling logs

Currently, our Parameter Group looks like next:

```
...
    "MasterDBParamGroup": {
      "Type": "AWS::RDS::DBParameterGroup",
      "Properties": {
        "Description": "Master Database Parameter Group",
        "Family": "mariadb10.0",
        "Parameters" : {
          "net_read_timeout": 60
        },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "master-db-params"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },
...
```

RDS has four logs types:

- audit
- error
- general
- slowquery

To enable them need to update such parameters:

- `slow_query_log`
- `general_log`
- `log_error` (enabled by default)

Also good to update those:

- `long_query_time` – time in seconds after which a request will be logged to the Slow log
- `log_output` – set to _FILE to enable export to the_ CloudWatch Logs

At first – let’s do it via AWS UI, and then will update a CloudFormation template.

Find an instance’s Parameter Group:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105835.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105835.png)

Add the necessary options:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105955.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105955.png)

Check them:

```
$ aws rds --profile bm-backend describe-db-parameters --db-parameter-group-name mobilebackend-dev-masterdbparamgroup-258i35vgxvdi | grep -A1 'slow_query_log\|general_log\|log_error'

"ParameterName": "general_log",
"ParameterValue": "1",

--

"ParameterName": "general_log_file",
"ParameterValue": "/rdsdbdata/log/general/mysql-general.log",

--

"ParameterName": "log_error",
"ParameterValue": "/rdsdbdata/log/error/mysql-error.log",

--

"ParameterName": "slow_query_log",
"ParameterValue": "1",

--

"ParameterName": "slow_query_log_file",
"ParameterValue": "/rdsdbdata/log/slowquery/mysql-slowquery.log",
```

Logs have to appear in the _Logs & events_.

Also – do not forget to set `log_output` to the _FILE_ type (default value is _TABLE_):

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_110859.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_110859.png)

Save changes.

### CloudWatch Logs export

Go to an RDS instance and click _Modify_:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105659.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_105659.png)

List to the bottom and chose logs to be exported to the CloudWatch:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111710.png "AWS: RDS logs, export to the CloudWatch Logs and CloudFormation template")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111710.png)

Press _Continue_, check the list of settings to be updated, click _Modify DB instance_:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111753.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111753.png)

Wait a couple of minutes:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111819.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_111819.png)

Go to the CloudWatch Logs – and voila:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_112043.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_112043.png)

Logs data:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_112231.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_112231.png)

### CloudFormation template

The last thing is to update our CloudFormation template to apply changes to all servers.

Here we need to update the [`AWS::RDS::DBParameterGroup`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-dbparametergroup.html) resource and add the following parameters:

```
...
    "MasterDBParamGroup": {
      "Type": "AWS::RDS::DBParameterGroup",
      "Properties": {
        "Description": "Master Database Parameter Group",
        "Family": "mariadb10.0",
        "Parameters" : {
          "log_output": "FILE",
          "general_log": 1,
          "slow_query_log": 1,
          "long_query_time": 10,
          "log_queries_not_using_indexes": 1,
          "net_read_timeout": 60
        },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "master-db-params"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },
...
```

Next – enable the Export: update the [`AWS::RDS::DBInstance`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html) resource and add the [`EnableCloudwatchLogsExports`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html#cfn-rds-dbinstance-enablecloudwatchlogsexports) with logs list to be exported – _error_, _general_, _slowquery_:

```
...
    "DB1MasterRDS" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        "DBInstanceIdentifier" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "db1-master-rds"] ] },
        "MasterUsername" : { "Ref" : "DB1RootUser" },
        "MasterUserPassword" : { "Ref" : "DB1RootPassword" },
        "AllocatedStorage" : { "Ref" : "DB1AllocatedStorage" },
        "Engine" : "mariadb",
        "StorageType": "gp2",
        "AutoMinorVersionUpgrade": false,
        "PubliclyAccessible" : false,
        "DBInstanceClass" : { "Ref" : "DB1MasterDBInstanceClass" },
        "DBSubnetGroupName" : { "Ref" : "DB1SubnetGroup" },
        "DBParameterGroupName" : {"Ref" : "MasterDBParamGroup" },
        "EnableCloudwatchLogsExports": [
              "error",
              "general",
              "slowquery"
            ],
        "BackupRetentionPeriod": {"Ref" : "BackupRetentionPeriod" },
        "PreferredBackupWindow": {"Ref" : "PreferredBackupWindow" },
        "VPCSecurityGroups" : [ { "Fn::GetAtt" : [ "MasterDBSecurityGroup", "GroupId" ] } ],
        "MultiAZ": {"Ref" : "MultiAZ" },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "db1-master-rds"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      },
      "DeletionPolicy" : "Snapshot"
    },
...
```

Run a stack update:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_115129.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_115129.png)

Check log groups:

[![](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_120934.png "AWS: RDS логи, сбор в CloudWatch Logs и CloudFormation")](https://rtfm.co.ua/wp-content/uploads/2019/03/Screenshot_20190307_120934.png)

Done.


### Similar posts
  
- <small>05/29/2018</small> [MariaDB: AWS RDS – Lost connection to MySQL Server During Query](https://rtfm.co.ua/mariadb-aws-rds-lost-connection-to-mysql-server-during-query/)<small> (0)</small>  
- <small>10/27/2017</small> [AWS: CloudFormation – шаблон для RDS PostgreSQL](https://rtfm.co.ua/aws-cloudfromation-shablon-dlya-rds-postgresql/)<small> (0)</small>  
- <small>03/02/2018</small> [AWS: RDS – Connection refused и StorageFull](https://rtfm.co.ua/aws-rds-connection-refused-i-storagefull/)<small> (0)</small>  
- <small>05/17/2018</small> [AWS: GDPR и RDS – миграция на encrypted RDS](https://rtfm.co.ua/aws-gdpr-i-rds-migraciya-na-encrypted-rds/)<small> (0)</small>