Fluentd Use Case

exec 플러그인 활용하기

in_exec 플러그인을 활용하면 사용자가 수집하고자 하는 정확한 정보를 만들어내서 수집할 수 있다.

linux_free

linux 모니터링 명령어인 free 의 결과를 주기적으로 수집해본다.

데이터 소스 스크립트

vi /app/fluentd/scripts/free.sh

#!/bin/bash

DATE=date "+%Y-%m-%d %H:%M:%S"
DATA=free -k | grep "Mem:" | awk '{print $2,$3,$4,$5,$6,$7}'

echo $DATA $DATE

fluentd config

# linux_free
## Input

    @type exec
    command /app/fluentd/scripts/free.sh

        @type regexp
        expression /^(?\d+)\s+(?\d+)\s+(?\d+)\s+(?\d+)\s+(?\d+)\s+(?\d+)\s+(?.*)$/

    tag "exec.free.#{Socket.gethostname}"
    run_interval 3s
    time_key time
    time_format %Y-%m-%d %H:%M:%S

## Filter

    @type record_transformer

        hostname "#{Socket.gethostname}"

## Output

    @type copy
    #
    #   @type stdout
    #   include_tag_key true
    #   include_time_key true
    #

        @type elasticsearch
        hosts 192.168.179.81:9200,192.168.179.82:9200,
        logstash_format true
        logstash_prefix linux_free
        logstash_dateformat %Y%m%d
        include_tag_key true
        tag_key @log_name
        flush_interval 3s

linux_vmstat

linux 모니터링 명령어인 vmstat의 결과를 주기적으로 수집해본다.

데이터 소스 스크립트

vi /app/fluentd/scripts/vmstat.sh

#!/bin/bash

vmstat -t | grep -v "^procs" | grep -v -P "^\s+r"

fluentd config

# linux_vmstat
## Input
<source>
    @type exec
    command /app/fluentd/scripts/vmstat.sh
    <parse>
        @type regexp
        expression /^(?<procs_r>\d+)\s+(?<procs_b>\d+)\s+(?<memory_swpd>\d+)\s+(?<memory_free>\d+)\s+(?<memory_buff>\d+)\s+(?<memory_cache>\d+)\s+(?<swap_si>\d+)\s+(?<swap_so>\d+)\s+(?<io_bi>\d+)\s+(?<io_bo>\d+)\s+(?<system_in>\d+)\s+(?<system_cs>\d+)\s+(?<cpu_us>\d+)\s+(?<cpu_sy>\d+)\s+(?<cpu_id>\d+)\s+(?<cpu_wa>\d+)\s+(?<cpu_st>\d+)\s+(?<time>.*)$/
    </parse>
    tag "exec.vmstat.#{Socket.gethostname}"
    run_interval 3s
    time_key time
    time_format %Y-%m-%d %H:%M:%S
</source>

## Filter
<filter exec.vmstat.**>
    @type record_transformer
    <record>
        hostname "#{Socket.gethostname}"
    </record>
</filter>

## Output
<match exec.vmstat.**>
    @type copy
    #<store>
    #   @type stdout
    #   include_tag_key true
    #   include_time_key true
    #</store>
    <store>
        @type elasticsearch
        hosts 192.168.179.81:9200,192.168.179.82:9200,
        logstash_format true
        logstash_prefix linux_vmstat
        logstash_dateformat %Y%m%d
        include_tag_key true
        tag_key @log_name
        flush_interval 3s
    </store>
</match>

linux_who

linux 모니터링 명령어인 who의 결과를 주기적으로 수집해본다.

데이터 소스 스크립트

vi /app/fluentd/scripts/who.sh

#!/bin/bash

who | awk '{print $0, strftime("%Y-%m-%d %H:%M:%S")}'

fluentd config

# linux_who
## Input
<source>
    @type exec
    command /app/fluentd/scripts/who.sh
    <parse>
        @type regexp
        expression /^(?<NAME>\S+)\s+(?<LINE>\S+)\s+(?<TIME>\S{3}\s\d+\s\d{2}:\d{2}+)\s+(?<COMMENT>\S+)\s+(?<time>.*)$/
    </parse>
    tag "exec.who.#{Socket.gethostname}"
    run_interval 3s
    time_key time
    time_format %Y-%m-%d %H:%M:%S
</source>

## Filter
<filter exec.who.**>
    @type record_transformer
    <record>
        hostname "#{Socket.gethostname}"
    </record>
</filter>

## Output
<match exec.who.**>
    @type copy
    #<store>
    #   @type stdout
    #   include_tag_key true
    #   include_time_key true
    #</store>
    <store>
        @type elasticsearch
        hosts 192.168.179.81:9200,192.168.179.82:9200,
        logstash_format true
        logstash_prefix linux_who
        logstash_dateformat %Y%m%d
        include_tag_key true
        tag_key @log_name
        flush_interval 3s
    </store>
</match>