ELK + MetricBeat System Monitoring

Architecture

아래와 같은 구성도로 Monitoring System을 구축한다.

ELK MetricBeat Architecture

총 세 대의 머신으로 ELK 클러스터를 구성하며 metricbeat는 agent로서 서버 클러스터에는 설치될 필요가 없다.

hostname	components
node1.elk	elasticsearch, logstash, kibana
node2.elk	elasticsearch
node3.elk	elasticsearch

설치

사전 준비

JDK
- 1.8 이상
kernel
- vm.max_map_count = 262144
user limit
- elastic soft nofile 65536
- elastic hard nofile 65536
system user (elastic)
app, data directory
- app
- /app/elasticsearch
- /app/logstash
- /app/kibana
- data
- /data/elasticsearch

ElasticSearch

ElasticSearch 바이너리 다운로드

6.4.2 버전 기준으로 진행

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.tar.gz
tar -xvzf elasticsearch-6.4.2.tar.gz -C /app/elasticsearch
mv /app/elasticsearch/elasticsearch-6.4.2 /app/elasticsearch/6.4.2
ln -s /app/elasticsearch/6.4.2/* /app/elasticsearch/.

환경 설정

시스템 프로파일 및 실행 스크립트

시스템 프로파일과 실행 스크립트는 node1,2,3 모두 공통

vi ~/elastic-cmd.sh

#!/bin/bash

ES_HOME=/app/elasticsearch
ES_PROC=$ES_HOME/bin/elasticsearch
ES_PID=$ES_HOME/elasticsearch_pid

if [ "$1" = "start" ];then
        if [ -e $ES_PID ];then
                echo "Start up failed, please check if ElasticSearch (PID:cat $ES_PID) is running."
        else
                echo "$ES_PROC -p $ES_PID &"
                $ES_PROC -p $ES_PID &
        fi
elif [ "$1" = "stop" ];then
        if [ -e $ES_PID ];then
                PID=cat $ES_PID
                kill $PID
                while ps -p $PID > /dev/null; do sleep 1;done
                echo "ElasticSearch successfully stopped."
        else
                echo "Stop Failed, please check if ElasticSearch was already stopped."
        fi
else
        echo "Undefined cmd"
fi

vi ~/.bash_profile

# JDK ENV
export JAVA_HOME=/app/jdk
export PATH=$JAVA_HOME/bin:$PATH

# ELASTICSEARCH ENV
export PATH=/app/elasticsearch/bin:$PATH

# ELASTICSEARCH RUN SCRIPT
export PATH=/home/elastic:$PATH

# ALIASES
alias escfg='cd /app/elasticsearch/config'
alias escfgv='vi /app/elasticsearch/config/elasticsearch.yml'
alias eslog='cd /data/elasticsearch/logs'
alias eslogt='tail -128f /data/elasticsearch/logs/elasticsearch-cluster.log'

elasticsearch.yml

elasticsearch.yml 파일은 node1과 node2,3 는 다르게 설정한다.

vi /app/elasticsearch/config/elasticsearch.yml

node1

cluster.name: elk-sysmon
node.name: ${HOSTNAME}
node.data: false
node.master: true
node.ingest: true
http.enabled: true
http.port: 9200
transport.tcp.port: 9300
path.data: /data/elastic/data
path.logs: /data/elastic/logs
network.host: _site_
discovery.zen.ping.unicast.hosts: ["node1.elk","node2.elk", "node3.elk"]
discovery.zen.minimum_master_nodes: 2

node2,3

cluster.name: elk-sysmon
node.name: ${HOSTNAME}
node.data: true
node.master: true
node.ingest: true
http.enabled: true
http.port: 9200
transport.tcp.port: 9300
path.data: /data/elastic/data
path.logs: /data/elastic/logs
network.host: _site_
discovery.zen.ping.unicast.hosts: ["node1.elk","node2.elk", "node3.elk"]
discovery.zen.minimum_master_nodes: 2

Logstash

Logstash 바이너리 다운로드

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz
tar -xvzf logstash-6.4.2.tar.gz -C /app/logstash/.

환경 설정

vi /app/logstash/6.4.2/config/logstash-metricbeat.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://node1.elk:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

vi /app/logstash/6.4.2/config/pipeline.yml

- pipeline.id: metricbeat
  path.config: "/app/logstash/6.4.2/config/logstash-metricbeat.conf"
  pipeline.workers: 2

Kibana

Kibana 바이너리 다운로드

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz
tar -xvzf kibana-6.4.2-linux-x86_64.tar.gz -C /app/kibana/.
mv /app/kibana/kibana-6.4.2-linux-x86_64 /app/kibana/6.4.2
ln -s /app/kibana/6.4.2/* /app/kibana/.

환경 설정

vi /app/kibana/config/kibana.yml

server.port: 5601
server.host: "node1.elk"
server.name: "node1.elk"
elasticsearch.url: "http://node1.elk:9200"
kibana.index: ".kibana"
elasticsearch.requestTimeout: 60000
logging.dest: /data/elastic/logs/kibana.log

실행

ElasticSearch

@node1.elk, node2.elk, node3.elk

elastic-cmd.sh start

Logstash

@node1.elk

/app/logstash/6.4.2/bin/logstash &

Kibana

@node1.elk

/app/kibana/bin/kibana &

MetricBeat

MetricBeat 다운로드 및 설치

리소스 정보를 수집하고자 하는 서버에 설치

wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.4.2-linux-x86_64.tar.gz
tar -xvzf metricbeat-6.4.2-linux-x86_64.tar.gz

환경 설정 및 dashboard 설치

아래와 같이 vi metricbeat-6.4.2-linux-x86_64/metricbeat.yml 파일을 열어 아래 부분을 찾아 elasticsearch 호스트를 변경해준다.

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  hosts: ["node1.elk:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

이이서 kibana에서 보여질 dashboard template 을 생성한다.

./metricbeat setup --dashboards

dashboard 설치가 완료되면 vi metricbeat.yml 파일을 생성해 다음과 같이 입력한다.

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    #- core           # Per CPU core usage
    #- diskio         # Disk IO
    #- filesystem     # File system usage for each mountpoint
    #- fsstat         # File system summary metrics
    #- raid           # Raid
    #- socket         # Sockets and connection info (linux only)
  enabled: true
  period: 10s
  processes: ['.*']

  # Configure the metric types that are included by these metricsets.
  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.

#----------------------------- Logstash output ---------------------------------
output.logstash:
  # Boolean flag to enable or disable the output module.
  #enabled: true

  # The Logstash hosts
  hosts: ["node1.elk:5044"]

  # Number of workers per Logstash host.
  worker: 2

./metricbeat -c metricbeat.yml 명령으로 metricbeat 를 실행하면 끝.

기본 템플릿 만으로 다음과 같이 볼 수 있고 추가 개발하거나 커스터마이징 해서 보고자 하는 내용을 더 추가하면 된다.

ELK MetricBeat Dashboard

[ElasticSearch] ELK + MetricBeat System Monitoring

ELK + MetricBeat System Monitoring

Architecture

설치

사전 준비

ElasticSearch

ElasticSearch 바이너리 다운로드

환경 설정

시스템 프로파일 및 실행 스크립트

elasticsearch.yml

node1

node2,3

Logstash

Logstash 바이너리 다운로드

환경 설정

Kibana

Kibana 바이너리 다운로드

환경 설정

실행

ElasticSearch

Logstash

Kibana

MetricBeat

MetricBeat 다운로드 및 설치

환경 설정 및 dashboard 설치

You Might Also Like

Ollama ✕ WSL 2 ✕ VSCode Code GPT

re:Invent 2022 참가 후기

[windows] Lenovo Thinkpad X390