가용성을 고려한 Hadoop Cluster 설치

Namenode와 Resourcemanager를 active node,standby node로 각각 두 벌씩 구성함으로써 고가용성을 고려한 Hadoop Cluster를 설치한다.

사전 준비

의존 패키지 설치

namenode와 resourcemanager가 장애 상황에서 standby -> active 상태로 전환될 때에 fuser 명령을 이용해서 장애노드의 프로세스들을 정리하므로 psmisc 패키지를 설치한다.

yum -y install psmisc

JDK

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html 페이지에서 JDK 8 버전 바이너리 다운로드하고 /app/jdk 경로에 압축해제 하여 사용

ls -l /app/jdk

total 0
drwxr-xr-x 7 bigdata bigdata 245 Jul  7  2018 1.8.0_181
lrwxrwxrwx 1 bigdata bigdata  15 Oct  8 18:03 bin -> ./1.8.0_181/bin
lrwxrwxrwx 1 bigdata bigdata  19 Oct  8 18:03 include -> ./1.8.0_181/include
lrwxrwxrwx 1 bigdata bigdata  26 Oct  8 18:03 javafx-src.zip -> ./1.8.0_181/javafx-src.zip
lrwxrwxrwx 1 bigdata bigdata  15 Oct  8 18:03 jre -> ./1.8.0_181/jre
lrwxrwxrwx 1 bigdata bigdata  15 Oct  8 18:03 lib -> ./1.8.0_181/lib
lrwxrwxrwx 1 bigdata bigdata  15 Oct  8 18:03 man -> ./1.8.0_181/man
lrwxrwxrwx 1 bigdata bigdata  19 Oct  8 18:03 src.zip -> ./1.8.0_181/src.zip

ZooKeeper 설치

zookeeper 설치 자료 참고
https://oboki.net/workspace/bigdata/zookeeper/zookeeper-3-x-%EC%84%A4%EC%B9%98/

/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.179.80 node1.df node1.zk
192.168.179.81 node2.df node2.zk
192.168.179.82 node3.df node3.zk

사용자 생성

useradd -u 8020 hadoop -g bigdata
passwd hadoop

사용자 프로파일

vi ~/.bash_profile

## Java
export JAVA_HOME=/app/jdk
export PATH=$JAVA_HOME/bin:$PATH

## Hadoop
export HADOOP_HOME=/app/hadoop
export PATH=$HADOOP_HOME/bin:$PATH:$HADOOP_HOME/sbin:$PATH

## User
set -o vi

Directory 생성

mkdir -p /data/hadoop/namenode /data/hadoop/datanode /data/hadoop/journalnode /data/hadoop/yarn
chown hadoop:bigdata /data/hadoop -R
mkdir /app/hadoop
chown hadoop:bigdata /app/hadoop

SSH 공개키 생성 및 교환

마스터 노드는 모든 슬레이브에 ssh 접속이 가능해야 한다. 모든 노드에서 hadoop 계정으로 아래 명령 수행한다.

ssh-keygen -t rsa -P ""
ssh node1.df cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh node2.df cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh node3.df cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Hadoop 설치

본격적으로 hadoop 설치. 여기서부터는 전부 hadoop 계정에서 진행한다.

바이너리 압축 해제 및 링크 설정

wget http://mirror.navercorp.com/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz
tar -xvzf hadoop-2.9.2.tar.gz -C /app/hadoop/.
mv /app/hadoop/hadoop-2.9.2/ /app/hadoop/2.9.2
find /app/hadoop/2.9.2 -mindepth 1 -maxdepth 1 -type d | xargs -I {} ln -s {} /app/hadoop/.

Hadoop Cluster 환경설정

다음과 같은 파일을 수정하여 Hadoop Cluster를 설정한다.

slaves
core-site.xml
hdfs-site.xml
yarn-site.xml
hadoop-env.sh

cd $HADOOP_HOME/hadoop/etc

slaves

vi slaves

node1.df
node2.df
node3.df

core-site.xml

vi core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop-cluster</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/tmp/hadoop</value>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>node1.zk:2181,node2.zk:2181,node3.zk:2181</value>
    </property>
    <property>
        <name>ipc.client.connect.max.retries</name>
        <value>100</value>
        <description>Indicates the number of retries a client will make to establish a server connection.</description>
    </property>
    <property>
        <name>ipc.client.connect.retry.interval</name>
        <value>10000</value>
        <description>Indicates the number of milliseconds a client will wait for before retrying to establish a server connection.</description>
    </property>
    <property>
        <name>hadoop.proxyuser.super.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.super.groups</name>
        <value>*</value>
    </property>
     <property>
        <name>io.comporession.codecs</name>
        <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZipCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value> 
    </property>
    <property>
        <name>io.compression.codec.lzo.class</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property>
    <property>
        <name>fs.inmemory.size.mb</name>
        <value>200</value>
    </property>
    <property>
        <name>io.sort.factor</name>
        <value>100</value>
    </property>
    <property>
        <name>io.sort.mb</name>
        <value>500</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
</configuration>

hdfs-site.xml

vi hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/hadoop/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.name.dir</name>
        <value>/data/hadoop/datanode</value>
    </property>
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data/hadoop/journalnode</value>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-cluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.hadoop-cluster</name>
        <value>nn1,nn2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.hadoop-cluster.nn1</name>
        <value>node1.df:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.hadoop-cluster.nn2</name>
        <value>node2.df:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.hadoop-cluster.nn1</name>
        <value>node1.df:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.hadoop-cluster.nn2</name>
        <value>node2.df:50070</value>
    </property>
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://node1.df:8485;node2.df:8485;node3.df:8485/hadoop-cluster</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.hadoop-cluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence(hadoop:22)</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadoop/.ssh/id_rsa</value>
    </property>
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.support.append</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.support.broken.append</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.block.size</name>
        <value>268435456</value>
    </property>
    <property>
        <name>dfs.namenode.handler.count</name>
        <value>20</value>
    </property>
    <property>
        <name>dfs.datanode.handler.count</name>
        <value>20</value>
    </property>
    <property>
        <name>tasktracker.http.threads</name>
        <value>400</value>
    </property>
    <property>
        <name>dfs.hosts.exclude</name>
        <value>/app/hadoop/etc/hadoop/dfs.exclude</value>
        <final>true</final>
    </property>
</configuration>

yarn-site.xml

vi yarn-site.xml

<?xml version="1.0"?>

<configuration>

    <property>
        <name>yarn.acl.enable</name>
        <value>0</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-service</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-service.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffledHandler</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/data/hadoop/yarn/nm-local-dir</value>
    </property>
    <property>
        <name>yarn.resourcemanager.fs-state.uri</name>
        <value>/data/hadoop/yarn/system/rmstore</value>
    </property>
    <!-- for YARN RM HA -->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true<value>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yarn-cluster</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>node1.df</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>node2.df</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>node1.df:8050</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>node2.df:8050</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>node1.df:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>node2.df:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>node1.zk:2181,node2.zk:2181,node3.zk:2181</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
        <description>Enable automatic failover; By default, it si enabled only when HA is enabled.</description>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>node1.df:8025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>node2.df:8025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>node1.df:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>node2.df:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.connect.retry-interval.ms</name>
        <value>2000</value>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
    </property>
    <property>
        <name>yarn.nodemanager.disk-health-checker.max-disk-utilizaion-per-disk-percentage</name>
        <value>98.5</value>
    </property>
    <!--for Spark  -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <!--for App Memery  -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>
    <!-- for Spark Dynamic Setting -->
    <!-- spark.executor.cores * spark.executor.instances  -->
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>2</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>2</value>
    </property>
</configuration>

hadoop-env.sh

vi hadoop-env.sh

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/app/jdk
export HADOOP_SSH_OPTS="-p 22"

Namenode format 및 Hadoop Cluster 실행

Namenode format

모든 노드에서 아래 명령 수행

touch /app/hadoop/etc/hadoop/dfs.exclude
hadoop-daemon.sh start journalnode

Primary Namenode 에서 아래 명령 수행

hdfs namenode -format
scp -r /data/hadoop/namenode/current node2.df:/data/hadoop/namenode/.
hdfs zkfc -formatZK