Monitoring single-node Ceph cluster using Prometheus & Grafana on AWS
Tags: devops, ceph, grafana, prometheus
Published: 2020-03-02 17:39:44

### Basic Information:

**Ceph-**
     Ceph is a free-software storage platform, implements object storage on a single distributed computer cluster, and provides interfaces for object-, block- and file-level storage. Ceph aims primarily for completely distributed operation without a single point of failure, and freely available. 

**Prometheus-**
     Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community.

**Grafana-**
     Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. It allows to create, explore, and share dashboards with your team and foster a data driven culture.
_________________________________

###Integration of Ceph with Prometheus and Grafana on AWS:

Let's divide this into __3 tasks__ so as to get our desired monitoring tool for ceph
__1. Installation & configuration of Ceph__
__2. Installing & configuring Prometheus__
__3. Installing & configuring Grafana__

__NOTE:__ I'm assuming y'all know the basics of AWS. You need to launch an instance with few prerequisites:
_1. Add 3 volumes for osds._
_2. Security groups. (Allow All_TCP, HTTP and HTTPS from anywhere)_
_____

####1. Installation & configuration of Ceph:
####ssh-keygen:
`[root@ip-10-0-0-100 ~]# vim /etc/ssh/sshd_config`
Make these changes in the file:
```
PermitRootLogin yes
PasswordAuthentication yes
```
`[root@ip-10-0-0-100 ~]# systemctl restart sshd`

`[root@ip-10-0-0-100 ~]# ssh-keygen`

`[root@ip-10-0-0-100 ~]# ssh-copy-id root@ip-10-0-0-100`

####Configure Ceph repo:

`[root@ip-10-0-0-100 ~]# vim /etc/yum.repos.d/ceph.repo`
```yml
[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-nautilus/el7/$basearch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-nautilus/el7/noarch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-nautilus/el7/SRPMS
enabled=0
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
```

####Install ceph-deploy
`[root@ip-10-0-0-100 ~]# yum install ceph-deploy -y`

`[root@ip-10-0-0-100 ~]# mkdir ceph-deploy`
`[root@ip-10-0-0-100 ~]# cd ceph-deploy`

####Create the cluster:

`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy new {initial-monitor-node(s)}`
_example: [root@ip-10-0-0-100 ceph-deploy]# ceph-deploy new ip-10-0-0-100_

####Install Ceph packages:

`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy install {node(s)}`
_example: [root@ip-10-0-0-100 ceph-deploy]# ceph-deploy install ip-10-0-0-100_

####Deploy the initial monitor(s) and gather the keys

`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy gatherkeys node`
_example: [root@ip-10-0-0-100 ceph-deploy]# ceph-deploy gatherkeys ip-10-0-0-100_ 
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy mon create-initial`

####Create admin
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy admin ip-10-0-0-100`
####Create Mgr
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy mgr create ip-10-0-0-100`

#### Create Mds
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy mds create ip-10-0-0-100`

####Create Osds

`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy osd create --data /dev/xvdb ip-10-0-0-100`
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy osd create --data /dev/xvdc ip-10-0-0-100`
`[root@ip-10-0-0-100 ceph-deploy]# ceph-deploy osd create --data /dev/xvdd ip-10-0-0-100`

####Check your cluster's status
`[root@ip-10-0-0-100 ceph-deploy]# ceph -s`
```
 cluster:
    id:     5e4cfca2-43b5-13fd2-aee2b4808d95
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ip-10-0-0-100 (age 3m)
    mgr: ip-10-0-0-100(active, since 3m)
    osd: 3 osds: 3 up (since 3m), 3 in (since 7h)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 18 GiB / 21 GiB avail
    pgs:     

```

####2. Installation & configuration of Prometheus:

####Update System

`[root@ip-10-0-0-100 ~]# yum update -y`


####Download Prometheus package

Go to official Prometheus downloads [downloads](https://prometheus.io/download/) page, and copy the URL of Linux “tar” file.


`[root@ip-10-0-0-100 ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz`

####Configure Prometheus

#####Add a Prometheus user.

`[root@ip-10-0-0-100 ~]# useradd --no-create-home --shell /bin/false prometheus`


#####Create needed directories.

`[root@ip-10-0-0-100 ~]# mkdir /etc/prometheus`

`[root@ip-10-0-0-100 ~]# mkdir /var/lib/prometheus`


#####Change the owner of the above directories.

`[root@ip-10-0-0-100 ~]# chown prometheus:prometheus /etc/prometheus`

`[root@ip-10-0-0-100 ~]# chown prometheus:prometheus /var/lib/prometheus`


#####Now go to Prometheus downloaded location and extract it.

`[root@ip-10-0-0-100 ~]# tar -xvzf prometheus-2.16.0.linux-amd64.tar.gz`


#####Rename it as per your preference.

`[root@ip-10-0-0-100 ~]# mv prometheus-2.16.0.linux-amd64 prometheuspackage`

#####Copy “prometheus” and “promtool” binary from the “prometheuspackage” folder to “/usr/local/bin”.

`[root@ip-10-0-0-100 ~]# cp prometheuspackage/prometheus /usr/local/bin/`

`[root@ip-10-0-0-100 ~]# cp prometheuspackage/promtool /usr/local/bin/`

#####Change the ownership to Prometheus user.

`[root@ip-10-0-0-100 ~]# chown prometheus:prometheus /usr/local/bin/prometheus`

`[root@ip-10-0-0-100 ~]# chown prometheus:prometheus /usr/local/bin/promtool`

#####Copy “consoles” and “console_libraries” directories from the “prometheuspackage” to “/etc/prometheus folder”

`[root@ip-10-0-0-100 ~]# cp -r prometheuspackage/consoles /etc/prometheus`

`[root@ip-10-0-0-100 ~]# cp -r prometheuspackage/console_libraries /etc/prometheus`

#####Change the ownership to Prometheus user

`[root@ip-10-0-0-100 ~]# chown -R prometheus:prometheus /etc/prometheus/consoles`

`[root@ip-10-0-0-100 ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries`


####Add and modify Prometheus configuration file.

Configurations should be added to the  “/etc/prometheus/prometheus.yml”
Now we will create the prometheus.yml file.

`[root@ip-10-0-0-100 ~]# vim /etc/prometheus/prometheus.yml`
```yml
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_master'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
```
save and exit the file

#####Change the ownership of the file.

`[root@ip-10-0-0-100 ~]# chown prometheus:prometheus /etc/prometheus/prometheus.yml`

#####Configure the Prometheus Service File.

`[root@ip-10-0-0-100 ~]# vim /etc/systemd/system/prometheus.service`

#####Copy the following content to the file.
```
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
```
Save and the exit file.


#####Reload the systemd service.

`[root@ip-10-0-0-100 ~]# systemctl daemon-reload`

#####Start the Prometheus service.

`[root@ip-10-0-0-100 ~]# systemctl start prometheus`

#####Check service status.

`[root@ip-10-0-0-100 ~]# systemctl status prometheus`

#####Access Prometheus Web Interface
Use the following Url to access UI.
```
http://Server-IP:9090/graph
```
Then you can see the following interface.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wvty1206id8s4cav5ok9.png)

####3. Installation & configuration of Grafana:

#### Installing Grafana via YUM Repository

`[root@ip-10-0-0-100 ~]# vim /etc/yum.repos.d/grafana.repo`
Add these lines:
```
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
```
####Install Grafana
`[root@ip-10-0-0-100 ~]# yum install grafana -y`
 
####Enable Grafana Service
`[root@ip-10-0-0-100 ~]# systemctl start grafana-server`
`[root@ip-10-0-0-100 ~]# systemctl enable grafana-server`
`[root@ip-10-0-0-100 ~]# systemctl status grafana-server`

####Browse Grafana
```
http://Your Server IP or Host Name:3000/
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/krre0q8c03zcgtu1e11c.png)


####Mgr enable dashboard
`[root@ip-10-0-0-100 ~]# ceph mgr module enable dashboard --force`

####Mgr enable Prometheus
`[root@ip-10-0-0-100 ~]# ceph mgr module enable prometheus --force`

###Edit the ceph.conf
`[root@ip-10-0-0-100 ~]# vim ceph-deploy/ceph.conf`
Add:
```
[mon]
        mgr initial modules = dashboard

```
####Check Mgr Services
`[root@ip-10-0-0-100 ~]# ceph mgr services`
{
    "prometheus": "http://ip-10-0-0-100.ec2.internal:9283/"
}
__Copy this port number i.e. 9283__

####Edit the prometheus.yml
`[root@ip-10-0-0-100 ~]# vim /etc/prometheus/prometheus.yml`
Modify: __Paste 9283 here__
```
- targets: ['localhost:9283']
```
####Restart prometheus Service
`[root@ip-10-0-0-100 ~]# systemctl restart prometheus`

________________
###Now you have successfully installed prometheus and grafana for ceph monitoring
Let us begin the grafana setup now
a. Login to grafana with default username and password as admin
b. Create a datasource

--->Select Prometheus
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dqmn1fwy09y0p6k5dupw.png)

__Click on Save and Test__

c. Click on Create---> Import
 ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jtrnj3h4nxekcugme89l.png)
__Enter 7056 in:__
Grafana.com Dashboard
7056

d. Choose Name and Ceph Prometheus data source
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3s8abu7tojghovkxdhss.png)

###Finally we have successfully integrated our Ceph Cluster with Prometheus and Grafana!
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o4qw6ppk5raeb0ej4tjr.png)

Using this we can monitor our ceph storage graphically giving us much more and better insights!