Monitoring Server for Developers: ELK, Prometheus, Grafana, AlertManager.

2 min readFeb 2, 2022

Today we install Monitoring/Logging Server locally on VM with ELK (Elasticsearch, Logstash, Kibana), Prometheus, Grafana, AlertManager, Node Exporter using docker-compose and official images.

Developers and some professionals use local computer to deploy own infrastructure and test project or application locally. Here we install the most popular and powerful tools as a part of the local system. We neglect security and have direct access to resources which allows to make the development faster; that’s project is not for Production.

Install Linux on VM

Before start, you should have Linux with Docker on Oracle VM VirtualBox, VMware Workstation Player or even bare metal servers.

I don’t describe here how to do it assuming that you already have some servers for monitoring and logging. In my case, I’m running the Server on CentOS 8 and VMware and give it 8Gb RAM, 2 CPU, 30Gb disk.

Install Monitoring/Logging Server

I use the certain latest versions of docker images to be sure that would work properly on your system too. ELK 7.16, Prometheus 2.33.0, Grafana 8.3.4, AlertManager 0.23.0, Node exporter 0.18.1 but you can change them in docker-compose files.

sudo mkdir /opt/dev-monitor
sudo chmod og+wx /opt/dev-monitor
cd /opt/
git clone https://github.com/sergeyanover/dev-monitoring-server dev-monitor
cd dev-monitor
chmod a+x up-elk
chmod a+x up-prom-graf
chmod a+x up-elk-prom-graf
chmod a+x down-elk
chmod a+x down-prom-graf
chmod a+x down-elk-prom-graf
chmod o+w data/prometheus
sudo chown 472 data/grafana
sudo chgrp 472 data/grafana

To start/stop ELK, Prometheus and Grafana:

./up-elk-prom-graf
./down-elk-prom-graf

To start/stop ELK only:

./up-elk
./down-elk

To start/stop Prometheus and Grafana (without ELK):

./up-prom-graf
./down-prom-graf

In local environment for testing we can afford to open ports on every interface, not only localhost: Elasticsearch — 9200, Logstash — 5044, 5000, 9600 , Kibana — 5601, Prometheus — 9090, Grafana — 3000, AlertManager — 9093 without inbound proxy or firewall, Node Exporter — 9100.

I use domain.local as a name to resolve server’s IP address, so to connect:

Kibana: http://domain.local:5601 user: elastic password: mypass
Grafana: http://domain.local:3000 user: admin password: admin

Example with Node Exporter

Let’s monitor the server itself in Grafana. We should create a Dashboard. In Datasources we already have Prometheus — http://prometheus:9090 , you can change it in conf/grafana/datasources.yml or install new one via GUI Grafana. We have Node Exporter — http://node-exporter:9100 which provides some metrics of the host.

We should import Dashboard using JSON or just put ID from grafana.com https://grafana.com/grafana/dashboards/ . In GUI, just use 1 click — sign “plus” in left Menu and click Import. Try ID 1860 — Node Exporter Full and it works immediately.

Now, you can connect installed on servers agents like Filebeat, Metricbeat, Fluentd, StatsD etc. to your server using simple settings and open ports, that could be much faster and easier for Development purposes.

References

Monitoring Server for Developers: ELK, Prometheus, Grafana, AlertManager.

Written by Sergey Yanover