{"id":169,"date":"2018-07-17T13:30:23","date_gmt":"2018-07-17T04:30:23","guid":{"rendered":"https:\/\/dong1lkim.oboki.net\/?p=169"},"modified":"2019-09-01T22:21:42","modified_gmt":"2019-09-01T13:21:42","slug":"fluentd-%ec%84%a4%ec%b9%98","status":"publish","type":"post","link":"https:\/\/oboki.net\/workspace\/data-engineering\/fluentd\/fluentd-%ec%84%a4%ec%b9%98\/","title":{"rendered":"[Fluentd] Fluentd \uc124\uce58"},"content":{"rendered":"<h2>Fluentd \uc124\uce58\ud558\uae30<\/h2>\n<p>Linux \uc2dc\uc2a4\ud15c\uc5d0 \uc124\uce58\ud558\ub294 \uac83\uc744 \uae30\uc900\uc73c\ub85c \uc9c4\ud589.<br \/>\n\ud328\ud0a4\uc9c0 \uc124\uce58 or \uc18c\uc2a4 \ucef4\ud30c\uc77c \uc124\uce58 \ubaa8\ub450 \uac00\ub2a5\ud558\uc9c0\ub9cc \uc18c\uc2a4 \ucef4\ud30c\uc77c \ubc29\uc2dd\uc73c\ub85c \uc124\uce58 \uc9c4\ud589.<\/p>\n<h3>\uc758\uc874\uc131<\/h3>\n<ul>\n<li>zlib-devel<\/li>\n<li>openssl<\/li>\n<li>ruby<\/li>\n<\/ul>\n<h3>\uc124\uce58<\/h3>\n<ol>\n<li>System User Profile\n<pre><code>export LD_LIBRARY_PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/lib:$LD_LIBRARY_PATH\nexport PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/bin:$PATH\nexport RUBY_HOME=\/app\/fluentd\/ruby\/2.4.4\nexport PATH=$RUBY_HOME\/bin:$PATH\n<\/code><\/pre>\n<\/li>\n<li>Install openssl from source\n<ul>\n<li>openssl archive \ub2e4\uc6b4\ub85c\ub4dc<br \/>\n<code>wget https:\/\/www.openssl.org\/source\/openssl-1.1.1-pre6.tar.gz<\/code><\/li>\n<li>archive \uc555\ucd95\ud574\uc81c<br \/>\n<code>tar -xvzf openssl-1.1.1-pre6.tar.gz -C \/app\/fluentd\/ssl<\/code><\/li>\n<li>\uc124\uce58 \uacbd\ub85c\ub85c \uc774\ub3d9\ud558\uc5ec \uc124\uce58 \ud658\uacbd \uc124\uc815<br \/>\n<code>cd \/app\/fluentd\/ssl\/openssl-1.1.1-pre6<\/code><br \/>\n<code>.\/config --prefix=\/app\/fluentd\/ssl\/1.1.1-pre6 --openssldir=\/app\/fluentd\/ssl\/1.1.1-pre6<\/code><\/li>\n<li>Let&#8217;s make<br \/>\n<code>make &amp;&amp; make install<\/code><\/li>\n<li>\ubc14\uc774\ub108\ub9ac \uc2e4\ud589 \uacbd\ub85c \ubc0f \ub77c\uc774\ube0c\ub7ec\ub9ac \ub85c\ub4dc \uacbd\ub85c \uc124\uc815<br \/>\n<code>export LD_LIBRARY_PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/lib:$LD_LIBRARY_PATH<br \/>\nexport PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/bin:$PATH<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Install Ruby from source\n<ul>\n<li>Ruby archive \ub2e4\uc6b4\ub85c\ub4dc<br \/>\n<code>wget https:\/\/cache.ruby-lang.org\/pub\/ruby\/2.4\/ruby-2.4.4.tar.gz<\/code><\/li>\n<li>archive \uc555\ucd95\ud574\uc81c<br \/>\n<code>tar -xvzf ruby-2.4.4.tar.gz<\/code><\/li>\n<li>\uc124\uce58 \uacbd\ub85c\ub85c \uc774\ub3d9\ud558\uc5ec \uc124\uce58 \ud658\uacbd \uc124\uc815<br \/>\n<code>cd ruby-2.4.4<\/code><br \/>\n<code>.\/configure --prefix=\/app\/fluentd\/ruby\/2.4.4 --with-openssl-dir=\/app\/fluentd\/ssl\/1.1.1-pre6<\/code><\/li>\n<li>Let&#8217;s make!<br \/>\n<code>make &amp;&amp; make install<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Install fluentd using ruby gem\n<ul>\n<li>bundler \uc124\uce58<br \/>\n<code>gem install bundle<\/code><\/li>\n<li>fluentd \uc124\uce58 \uacbd\ub85c \uc774\ub3d9 \ubc0f \uc124\uce58<br \/>\n<code>cd \/app\/fluentd<\/code><br \/>\n<code>git clone https:\/\/github.com\/fluent\/fluentd.git<\/code><br \/>\n<code>cd fluentd<\/code><br \/>\n<code>bundle install<\/code><\/li>\n<li>\ud558\ub461 output plugin \uc124\uce58<br \/>\n<code>gem install fluent-plugin-webhdfs<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Run fluentd\n<ul>\n<li>config \ud30c\uc77c \uc815\uc758<br \/>\nfluent\ub294 \ud558\ub098\uc758 \uc124\uc815 \ud30c\uc77c\uc744 \ub85c\ub4dc\ud574\uc11c \uc2e4\ud589\ub41c\ub2e4.<br \/>\n<code>vi fluentd.conf<\/code><br \/>\n<code>&lt;source&gt;....&lt;\/source&gt;<\/code> \uad6c\ubb38 \ub0b4\uc5d0\uc11c input \ub370\uc774\ud130\ub97c \uc815\uc758\ud558\uace0, <code>&lt;match .. &gt;...&lt;\/match&gt;<\/code> \uad6c\ubb38\uc73c\ub85c output \ub370\uc774\ud130\ub97c \uc815\uc758\ud55c\ub2e4.<\/li>\n<li>input \ud0c0\uc785 \uc815\uc758\n<pre><code>&lt;source&gt;\n    @type http\n    port 8888\n&lt;\/source&gt;\n<\/code><\/pre>\n<\/li>\n<li>output \ud0c0\uc785 \uc815\uc758\n<pre><code>&lt;match hdfs.*.*&gt;\n    @type webhdfs\n    host bigdata-116\n    username fluentd\n    port 50070\n    path \"\/user\/fluentd\/log\/%Y%m%d_%H\/access.log.#{Socket.gethostname}\"\n    flush_interval 10s\n&lt;\/match&gt;\n<\/code><\/pre>\n<\/li>\n<li>fluent \uc2e4\ud589<br \/>\n<code>fluentd -c .\/fluent\/fluent.conf -vv &amp;<\/code><\/li>\n<li>\ub370\uc774\ud130 \ubc1c\uc0dd<br \/>\nfluent\ub97c http \ud0c0\uc785\uc73c\ub85c \uc2e4\ud589\ud588\uc73c\ubbc0\ub85c fluent\uac00 LISTENING \ud558\uace0 \uc788\ub294 \ud3ec\ud2b8\uc5d0 http \ub370\uc774\ud130\ub97c \uc804\ub2ec\ud568\uc73c\ub85c\uc368 \uc218\uc9d1\uc774 \ub418\ub294\uc9c0 \ud655\uc778\ud55c\ub2e4.<br \/>\n<code>curl -X POST -d 'json={\"action\":\"login\",\"user\":2}'   http:\/\/localhost:8888\/hdfs.access.test<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h3>Fluentd-ui<\/h3>\n<p>Fluentd \uacf5\ud648\uc5d0\uc11c \uad00\ub9ac\ud558\ub294 \ub4ef(?) \ud55c Fluentd \uad00\ub9ac WEB \uc778\ud130\ud398\uc774\uc2a4.<br \/>\nruby gem \uc73c\ub85c \uac04\ub2e8\ud558\uac8c plugin \uc124\uce58 \uac00\ub2a5.<br \/>\n<a href=\"https:\/\/docs.fluentd.org\/v0.12\/articles\/fluentd-ui\">https:\/\/docs.fluentd.org\/v0.12\/articles\/fluentd-ui<\/a><\/p>\n<h4>Installation of fluentd-ui using gem<\/h4>\n<p><code>gem install -V fluentd-ui<\/code><\/p>\n<h4>Run fluentd-ui<\/h4>\n<p><code>fluentd-ui start<\/code><br \/>\nit may return<\/p>\n<pre><code>[fluentd@bigdata-116 ~]$ fluentd-ui start\nPuma starting in single mode...\n* Version 3.11.4 (ruby 2.4.4-p296), codename: Love Song\n* Min threads: 0, max threads: 16\n* Environment: production\n* Listening on tcp:\/\/0.0.0.0:9292\nUse Ctrl-C to stop\n<\/code><\/pre>\n<h4>web \uc811\uc18d<\/h4>\n<ul>\n<li>\uc6f9 \ube0c\ub77c\uc6b0\uc800\uc5d0\uc11c \uc544\ub798 URL \uc785\ub825\n<pre><code>{\ud574\ub2f9 \uc11c\ubc84\uc758 IP \ud639\uc740 \ub3c4\uba54\uc778 \uc8fc\uc18c}:{LISTENING_PORT}\nhttp:\/\/bigdata-116:9292\nhttp:\/\/192.168.179.116:9292\n<\/code><\/pre>\n<\/li>\n<li>Default username &amp; password\n<pre><code>id: admin\npassword: changeme\n<\/code><\/pre>\n<\/li>\n<\/ul>\n<hr \/>\n<h2>\uad6c\uc131\ub3c4<\/h2>\n<p><img decoding=\"async\" src=\"https:\/\/docs.fluentd.org\/images\/fluentd_ha.png\"><\/p>\n<ul>\n<li>log forwarders<br \/>\nremote agent<\/p>\n<ul>\n<li>input plugin\uc740 \uc218\uc9d1\ud558\uace0\uc790 \ud558\ub294 \ub85c\uadf8\uc5d0 \uc54c\ub9de\uac8c \uc124\uc815<\/li>\n<li>output plugin\uc740 out_forward \ub9cc\uc744 \uc0ac\uc6a9\ud558\uc5ec log aggregators \uc5d0 \uc804\uc1a1<\/li>\n<\/ul>\n<\/li>\n<li>log aggregators\n<ul>\n<li>input plugin\uc740 in_forward \ub9cc\uc744 \uc0ac\uc6a9\ud558\uc5ec log forwarders \uc5d0\uc11c \uc18c\uc2a4\ub97c \uc804\ub2ec\ubc1b\uc74c<\/li>\n<li>\ud544\uc694\uc5d0 \ub530\ub77c filter\/parser\/buffer \ub808\uc774\uc5b4\ub97c \ucd94\uac00<\/li>\n<li>output plugin \uc740 \uc800\uc7a5\uc18c\uc5d0 \ub530\ub77c \uc54c\ub9de\uac8c \uc124\uc815<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr \/>\n","protected":false},"excerpt":{"rendered":"<p>Fluentd \uc124\uce58\ud558\uae30 Linux \uc2dc\uc2a4\ud15c\uc5d0 \uc124\uce58\ud558\ub294 \uac83\uc744 \uae30\uc900\uc73c\ub85c \uc9c4\ud589. \ud328\ud0a4\uc9c0 \uc124\uce58 or \uc18c\uc2a4 \ucef4\ud30c\uc77c \uc124\uce58 \ubaa8\ub450 \uac00\ub2a5\ud558\uc9c0\ub9cc \uc18c\uc2a4 \ucef4\ud30c\uc77c \ubc29\uc2dd\uc73c\ub85c \uc124\uce58 \uc9c4\ud589. \uc758\uc874\uc131 zlib-devel openssl ruby \uc124\uce58 System User Profile export LD_LIBRARY_PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/lib:$LD_LIBRARY_PATH export PATH=\/app\/fluentd\/ssl\/1.1.1-pre6\/bin:$PATH export RUBY_HOME=\/app\/fluentd\/ruby\/2.4.4 export PATH=$RUBY_HOME\/bin:$PATH Install openssl from source openssl archive \ub2e4\uc6b4\ub85c\ub4dc wget https:\/\/www.openssl.org\/source\/openssl-1.1.1-pre6.tar.gz archive \uc555\ucd95\ud574\uc81c tar -xvzf openssl-1.1.1-pre6.tar.gz -C \/app\/fluentd\/ssl \uc124\uce58 \uacbd\ub85c\ub85c [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[142,50],"class_list":["post-169","post","type-post","status-publish","format-standard","hentry","category-fluentd","tag-fluented","tag-50"],"_links":{"self":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/169","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/comments?post=169"}],"version-history":[{"count":11,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/169\/revisions"}],"predecessor-version":[{"id":1269,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/169\/revisions\/1269"}],"wp:attachment":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/media?parent=169"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/categories?post=169"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/tags?post=169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}