Laravel log stream to Kibana Elasticsearch Monolog
Tags: laravel, kibana, elasticsearch, learning
Published: 2023-12-08 10:52:00

ELK ( Elasticsearch, Logdash, and Kibana) is one of the popular log management platforms for log analysis. But for this project I will be using Laravel Monolog for handling the logs instead of Logdash.
Before getting into coding let’s know a little about the platforms.

## Monolog :
`Monolog ` is a PHP logging library that’s included in Laravel. This library is widely customizable and used because of the easy handle management and it provides build in handler that can easily help anyone to send logs to multiple destinations. 
Here’s `Monolog ` [source code](https://github.com/Seldaek/monolog) and [Laravel documentation](https://laravel.com/docs/10.x/logging#monolog-channel-customization) for monolog.
## Elasticsearch :
[`Elasticsearch `](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)is a free and [open-source](https://github.com/elastic/elasticsearch) analytics and search engine that returns and stores data in the JSON index format. Users may easily store, search and analyze huge quantities of data thanks to this engine. Because it uses an index rather than directly querying the text data, it is able to respond to search results in milliseconds. Simply put, `Elasticsearch `functions as a server that can handle JSON queries and return JSON information.
## Kibana:
`Kibana ` is a visual interface tool that lets you work with log data that has been mass-stored in Elasticsearch clusters to explore, visualize, and generate reports. `Kibana's ` primary functionalities are data analysis and querying. Furthermore, you may see data in several ways with `Kibana's ` visualization tools by utilizing heat maps, line graphs, pie charts, histograms, and geospatial support. You can look for root cause diagnostics in the data stored in Elasticsearch using a variety of techniques.
[Kibana documentation ](https://www.elastic.co/guide/en/kibana/current/index.html)

### Ok!! Let's start coding
## Step 01: Run Kibana and Elasticsearch with docker 
I am using docker to run `Laravel` , `Kibana `, `Elasticsearch` . I hope you already know how to run `Laravel `projects in docker. For `Kibana ` and `Elasticsearch `, Here's my docker-compose code: - 

```
 kibana: 
    image: kibana:8.11.1
    container_name : kibana_container
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      ELASTICSEARCH_URL: http://elasticsearch:9200
    networks:
      - laravel
  elasticsearch: 
    image: elasticsearch:8.11.1
    container_name: elasticsearch_container
    ports:
      - "9200:9200"
    environment:
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - discovery.type=single-node # To run single node 
      - xpack.security.enabled=false # To remove SSL complexity
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - "./elk-data:/usr/share/elasticsearch/data"
    
    networks:
      - laravel
```
PS : I am keeping all my container in `laravel `network.
Now `docker-compose up --build` to see `Kibana `and `Elasticsearch` is up and running.
Elasticsearch will run on : `http://localhost:9200/`
Kibana will run on : `http://localhost:5601/`
## Step 02: Setup `env ` file 
In .env add this lines: 
```
ELASTIC_HOST=elasticsearch_container:9200
ELASTIC_LOGS_INDEX=laravel_logs
LOG_CHANNEL=elastic
```
## Step 03: Create config file
Create a config file inside `config/elastic_log.php` . Write this in that file 

```
<?php

return [
    'host' => env('ELASTIC_HOST'),
    'index' => 'laravel_logs',
    'prefix' => 'api',
    'type' => '_doc',
];
```
## Step 04: Create ElasticLog Provider 
Create a Laravel provider with this command 

```
php artisan make:provider ElasticlogProvider 
```
and put this block of code inside `boot ` function 

```
        $index = rtrim(config('elastic_log.prefix'), '_') . '_' . config('elastic_log.index');
        $type = config('elastic_log.type');

        $this->app->bind(Client::class, function ($app) {
            return ClientBuilder::create()->setHosts([config('elastic_log.host')])->setHttpClient(new HttpClient(['verify' => false]))->build();
        });

        $this->app->bind(ElasticsearchFormatter::class, function ($app) use ($index, $type) {
            return new ElasticsearchFormatter($index, $type);
        });

        $this->app->bind(ElasticsearchHandler::class, function ($app) use ($index, $type) {
            return new ElasticsearchHandler($app->make(Client::class), [
                'index'        => $index,
                'type'         => $type,
                'ignore_error' => false,
            ]);
        });
```
## Step 04: Create Log channel 
In `config/logging.php ` add a new `monolog` channel named `elastic` under `channels ` array 
```
'elastic' => [
            'driver' => 'monolog',
            'handler' => ElasticsearchHandler::class,
            'level' => 'debug',
            'formatter' => ElasticsearchFormatter::class,
 ],
```
## Step 05: Create a custom command 
Create a Laravel custom command with 

```
php artisan make:command ElasticLogSetup
```
In `handle ` function add this :

```
$index = rtrim(config('elastic_log.prefix'), '_') . '_' . config('elastic_log.index');

if (!$this->client->indices()->exists(['index' => $index])) {
            $this->client->indices()->create([
                'index' => $index,
            ]);
        }
```
Make sure to change the signature of the command :

```
   protected $signature = 'elastic:log_setup';
```
Now run the command :

```
php artisan elastic:log_setup 
```
## Step 06: Add view data for viewing in Kibana
Check the Kibana container is running. If it's working go to 

```
http://localhost:5601/app/kibana#/management/kibana/index_patterns
```
And add the `api_laravel_logs` index into view 

![Kibana Dashboard Image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s9tebfuhcn50aaatjox9.png)

##Done 
Kibana now set up with Laravel project, You can stream logs with default Laravel `Log ` facade.

Thank you !!! 



### Resources :
* [ Elasticsearch: What It Is, How It Works, And What It’s Used For](Elasticsearch: What It Is, How It Works, And What It’s Used For)
* [What is Kibana Used for?](https://www.clariontech.com/platform-blog/what-is-kibana-used-for-10-important-features-to-know)
* [Laravel Log Management Using Elasticsearch and Kibana](https://tasmidur.medium.com/laravel-log-management-using-elasticsearch-and-kibana-10f0af42fd60#:~:text=The%20monolog%20provides%20an%20Elasticsearch,dashboards%20based%20on%20your%20logs.)