{"id":1724,"date":"2021-11-14T15:03:20","date_gmt":"2021-11-14T06:03:20","guid":{"rendered":"https:\/\/oboki.net\/workspace\/?p=1724"},"modified":"2021-11-19T15:07:34","modified_gmt":"2021-11-19T06:07:34","slug":"airflow-ha-on-docker","status":"publish","type":"post","link":"https:\/\/oboki.net\/workspace\/system\/docker\/airflow-ha-on-docker\/","title":{"rendered":"Airflow 2.x HA \uad6c\uc131 (3 node docker \ud658\uacbd)"},"content":{"rendered":"

airflow 2.0 \uc5d0\uc11c \uacf5\uc2dd\uc801(?)\uc73c\ub85c HA \ub97c \uc9c0\uc6d0\ud55c\ub2e4. 1.x \ubc84\uc804\uc5d0\uc11c\ub294 scheduler \ud504\ub85c\uc138\uc2a4\ub97c \ub3d9\uc2dc\uc5d0 \uc5ec\ub7ec\uac1c \uad6c\ub3d9\ud558\uba74 \uc791\uc5c5\uc774 \uc911\ubcf5 \uc2e4\ud589\ub420 \uc218 \uc788\ub294 \uc704\ud5d8\uc774 \uc788\uc5c8\uc11c \ubb34\uc870\uac74 single \ub85c \uc6b4\uc601\ud574\uc57c\ud588\uace0 \uc774 \ub54c\ubb38\uc5d0 \uac00\uc6a9\uc131\uc744 \ubcf4\uc7a5\ud560 \uc218 \uc5c6\uc5c8\ub294\ub370 2.x \uc5d0\uc11c\ubd80\ud130\ub294 row-level locking \uc744 \uc774\uc6a9\ud574\uc11c multiple scheduler \uc774\uc6a9\uc774 \uac00\ub2a5\ud558\ub3c4\ub85d \uac1c\uc120\uc774 \ub410\ub2e4. (\ub300\uc2e0 SKIP LOCKED<\/code> \ub610\ub294 NOWAIT<\/code> \uad6c\ubb38\uc744 \uc9c0\uc6d0\ud558\ub294 mysql8.0 \uc774\uc0c1\uc744 \uc368\uc57c\ud558\ub294 \uc81c\uc57d\uc870\uac74\uc774 \uc788\ub2e4.)<\/p>\n

\uc5b4\uca0b\ub4e0 1.x \uc5d0\uc11c\ub294 \uae30\ubcf8 \uae30\ub2a5\ub9cc\uc73c\ub85c\ub294 \uc644\ubcbd\ud55c \uace0\uac00\uc6a9\uc131 \uad6c\uc131\uc744 \uc9c0\uc6d0\ud558\uc9c0 \uc54a\uc558\uae30\ub54c\ubb38\uc5d0 \uc5b5\uc9c0\ub85c \uad6c\uc131\ud558\uc790\uba74 \ud560 \uc218 \uc788\uc5c8\ub358 HA \uad6c\uc131\uc744 \ubbf8\ub904\uc654\uc5c8\uc9c0\ub9cc.. 2.x \uc774 \ucd9c\uc2dc\ub418\uba74\uc11c \ub354\uc774\uc0c1 \ubbf8\ub8f0 \uc218 \uc5c6\uac8c \ub418\uc5c8\uace0 \uc6b0\ub9ac \ud300\uc5d0\uc11c\ub3c4 \uad00\ub828 PoC \ub97c \uc9c4\ud589\ud574\ubd24\ub2e4.<\/p>\n

\n

airflow 1.x \uc5d0\uc11c\ub294 teamclairvoyant \uc5d0\uc11c \uc81c\uacf5\ud558\ub294 \ud50c\ub7ec\uadf8\uc778\uc744 \ud65c\uc6a9\ud558\uba74 active-standby \ud615\ud0dc\uc758 \uac00\uc6a9\uc131\uc744 \ud655\ubcf4\ud560 \uc218 \uc788\ub2e4.
\nhttps:\/\/github.com\/teamclairvoyant\/airflow-scheduler-failover-controller<\/a><\/p>\n<\/blockquote>\n

\uc544\ubb34\ud2bc \uadf8\ub3d9\uc548 Webserver<\/code>, Scheduler(LocalExecutor)<\/code> \ub450 \ud504\ub85c\uc138\uc2a4\ub9cc\uc73c\ub85c \uc6b4\uc601\ud558\ub358 airflow \uc11c\ube44\uc2a4 \uad6c\uc131\uc774 \ucd5c\uc885\uc801\uc73c\ub85c \ub2e4\uc74c\uacfc \uac19\uc774 \ubcc0\ud588\ub2e4.<\/p>\n

\"\"<\/p>\n

docker-compose.yml<\/code>, airflow.cfg<\/code> \ub4f1 \uc804\uccb4 \uc124\uc815 \ud30c\uc77c\uc740 github.com\/oboki\/analytics-airflow\/tree\/HA<\/a> \uc5d0 \uc815\ub9ac\ud574\ub1a8\uace0 <\/p>\n

\uc704 \uad6c\uc131\uc774 \ub098\uc624\uae30 \uae4c\uc9c0 \uacaa\uc5c8\ub358 \uace0\ubbfc\uacfc \uc2dc\ud589\ucc29\uc624\uac00 \uba87\uac00\uc9c0 \uc788\uc5b4 \uac04\ub2e8\ud788(?) \uc801\uc5b4\ubd24\ub2e4.<\/p>\n

    \n
  1. Celery Broker \uc120\uc815 (SQLALchemy vs. Redis)<\/li>\n
  2. Redis HA \uad6c\uc131 \uc635\uc158 (Cluster vs. Sentinel)<\/li>\n
  3. \uba40\ud2f0 \ub178\ub4dc \ub3c4\ucee4 \ub124\ud2b8\uc6cc\ud06c \uc124\uc815<\/li>\n<\/ol>\n

    SQLAlchemy vs. Redis<\/h2>\n

    \uc544\uc9c1 \uc0ac\ub0b4\uc5d0\uc11c kubernetes<\/code> \ub97c \uc4f0\uc9c0\ub294 \ubabb\ud558\uae30 \ub54c\ubb38\uc5d0 celery worker<\/code> \ub97c \uc4f0\ub294\uac74 \uc815\ud574\uc838 \uc788\uc5c8\ub294\ub370 \uc774 celery \uc5d0\uc11c \uba54\uc2dc\uc9c0 \ube0c\ub85c\ucee4\ub85c \uc774\uc6a9\ud560 \uc218 \uc788\ub294 \uc635\uc158\uc774 rabbitMQ<\/code>, redis<\/code>, SQLAlchemy<\/code> \uc774\ub807\uac8c \uc788\ub2e4. \ud300 \ub0b4\uc5d0\uc11c \ud65c\uc6a9\ud558\ub294 rabbitMQ<\/code>, redis<\/code> \uac00 \ub530\ub85c \uc788\uc9c4 \uc54a\uc544\uc11c \ucd94\uac00 \uad6c\uc131\uc744 \ud574\uc57c\ud558\uae30\ub3c4 \ud588\uace0, \uc5b4\ucc28\ud53c meta DB \ub85c \ud65c\uc6a9\ud558\uace0\uc788\ub294 mysql<\/code>\uc744 queue \ub85c \uc7ac\ud65c\uc6a9\ud558\uba74 \ub354 \uc88b\uc744 \uac83 \uac19\uc544 \uba3c\uc800 SQLAlchemy<\/code> \ub97c \uc774\uc6a9\ud574\uc11c \uad6c\uc131\ud574\ubd24\uc5c8\ub2e4.<\/p>\n

    [celery]\nbroker_url = sqla+mysql:\/\/airflow:airflow@airflow-1:3306\/airflow<\/code><\/pre>\n
    \uc124\uc815\uc5d0\ub294 \ud06c\uac8c \uc5b4\ub824\uc6c0\ub3c4 \uc5c6\uc5c8\uace0 \uba87\uac00\uc9c0 \uac04\ub2e8\ud55c \uac00\uc6a9\uc131 \ud14c\uc2a4\ud2b8 \uc2dc\ub098\ub9ac\uc624\uc5d0\uc11c\ub3c4 \ubb38\uc81c\uac00 \uc5c6\uc5c8\uc9c0\ub9cc 3 \ubc84\uc804 \ubb38\uc11c\uc5d0 Experimental Status<\/em> \uac00 \ub208\uc5d0 \uac70\uc2ac\ub838\uace0 \ucd5c\uc2e0\ubc84\uc804 \ubb38\uc11c\uc5d0\uc11c\ub294 Using SQLAlchemy<\/em> \ud398\uc774\uc9c0\uac00 \uc874\uc7ac\ud558\uc9c0\ub3c4 \uc54a\ub294 \uc0c1\ud0dc\uc5ec\uc11c \uc544\ubb34\ub798\ub3c4 \uc801\uc6a9\ud558\uae30\uc5d0\ub294 \ubd80\ub2f4\uc2a4\ub7ec\uc6b4 \uac83 \uac19\uc544 \ub17c\uc758 \ub05d\uc5d0 sqla broker \ub294 \ud3ec\uae30\ud558\uae30\ub85c \ud588\ub2e4.<\/p>\n
    \n
    Brokers Using SQLAlchemy<\/a><\/p>\n<\/blockquote>\n
    \ub098\uba38\uc9c0 rabbitMQ<\/code>\uc640 redis<\/code> \uc911\uc5d0\uc11c\ub294 redis<\/code> \uac00 HA \uad6c\uc131\uc774 \uc880 \ub354 \ub0ab\ub2e4\uace0 \ud574\uc11c redis<\/code> \ub85c \uacb0\uc815\ud588\ub2e4.<\/p>\n
    Redis Cluster vs. Sentinel<\/h2>\n
    \ub808\ub514\uc2a4\uc5d0\uc11c\ub294 \uace0\uac00\uc6a9\uc131 \uad6c\uc131 \uc635\uc158\uc774 \ub450 \uac00\uc9c0 \uc788\ub294\ub370 active-active cluster \ub610\ub294 master-slave \uad6c\uc131\uc774\ub2e4. \uadf8 \uc911\uc5d0\uc11c \ucc98\uc74c\uc5d0\ub294 cluster\ub97c \uace0\ubbfc\ud588\uc5c8\ub294\ub370 quorum \uad6c\uc131\uc744 \uc704\ud55c 3 \ub178\ub4dc\uc5d0, \uac01 \ub178\ub4dc\ub9c8\ub2e4 master\u00d7slave pair \uc5d0\ub2e4\uac00 HAProxy \uae4c\uc9c0 \ud574\uc11c 3\uac1c \ud504\ub85c\uc138\uc2a4\ub97c \uad6c\uc131\ud574\uc57c\ud574\uc11c \ub808\ub514\uc2a4 \uad6c\uc131\ub9cc\uc744 \uc704\ud574\uc11c \ucd5c\uc18c 9\uac1c \ud504\ub85c\uc138\uc2a4\uac00 \ucd94\uac00\ub418\uc5b4\uc57c \ud55c\ub2e4.<\/p>\n
    SQLAlchemy \ube0c\ub85c\ucee4 \uad6c\uc131\uc5d0\uc11c\ub294 \ud544\uc694\ud55c \uc804\uccb4 \ud504\ub85c\uc138\uc2a4\uac00 6\uac1c \ubc16\uc5d0 \uc548 \ub410\ub294\ub370 \uc5ec\uae30\uc5d0 \uba54\uc2dc\uc9c0 \ube0c\ub85c\ucee4\ub85c\ub9cc 9\uac1c \ud504\ub85c\uc138\uc2a4\uac00 \ub354 \ucd94\uac00\ub41c\ub2e4\uace0 \ud558\ub2c8.. \ubc30\ubcf4\ub2e4 \ubc30\uaf3d\uc774 \ud070 \uac83 \uac19\uc544 cluster \uad6c\uc131\uc740 \ud3ec\uae30\ud588\ub2e4.<\/p>\n
    \ub2e4\ub978 \uc635\uc158\uc740, sentinel<\/code> \uc744 \uc774\uc6a9\ud55c active-standby \uad6c\uc131\uc778\ub370 celery \uc5d0\uc11c broker_url \ub85c sentinel url list \ub97c \uc9c0\uc815\ud560 \uc218 \uc788\uae30\ub54c\ubb38\uc5d0 \ubcc4\ub3c4\uc758 HAProxy\uac00 \ud544\uc694\uc5c6\uc5b4\uc11c \ucd5c\uc18c 5\uac1c \ud504\ub85c\uc138\uc2a4\ub85c \uad6c\uc131\ud560 \uc218 \uc788\ub2e4. slave \ub178\ub4dc\ub97c \ud558\ub098 \ub354 \ucd94\uac00\ud574\uc11c 6\uac1c \ud504\ub85c\uc138\uc2a4\ub85c \uad6c\uc131\ud558\uae34 \ud588\uc9c0\ub9cc, cluster \ubcf4\ub2e4 \uad6c\uc131\uc774 \ub2e8\uc21c\ud558\uace0 \uad00\ub9ac\uc5d0\ub3c4 \uc6a9\uc774\ud55c \uac83 \uac19\ub2e4.<\/p>\n
    airflow.cfg<\/code> \uc5d0\ub294 \ub2e4\uc74c\uacfc \uac19\uc740 \uc124\uc815\uc774 \ud544\uc694\ud558\ub2e4.<\/p>\n
    [celery]\nbroker_url = sentinel:\/\/airflow-1:26379\/0;sentinel:\/\/airflow-2\/0:26379;sentinel:\/\/airflow-3:26379\/0;\nresult_backend = db+mysql:\/\/airflow:airflow@airflow-1:3306\/airflow\n\n[celery_broker_transport_options]\nmaster_name = mymaster<\/code><\/pre>\n
    \uba40\ud2f0 \ub178\ub4dc \ub3c4\ucee4 \ub124\ud2b8\uc6cc\ud06c \uc124\uc815<\/h2>\n
    \ub85c\uceec \ud504\ub85c\uc138\uc2a4\ub85c \uad6c\ub3d9\ud558\ub358\uac78 \uc774\ubc88\uc5d0 \uc5c5\uadf8\ub808\uc774\ub4dc \ud558\uba74\uc11c \ub3c4\ucee4 \uae30\ubc18\uc73c\ub85c \uc6b4\uc601\ud558\uae30\ub85c \uacb0\uc815\ud588\uae30\ub54c\ubb38\uc5d0 docker-compose \ub85c \uad6c\uc131\ud574\ubd24\ub294\ub370 \uc6d0\uaca9\uc9c0 \uba38\uc2e0\uc758 \ucee8\ud14c\uc774\ub108\ub4e4 \uac04\uc758 \ud1b5\uc2e0\uc774 \uac00\ub2a5\ud558\ub3c4\ub85d \ud558\ub294 \ub124\ud2b8\uc6cc\ud06c \uc124\uc815\uc744 \ucc3e\ub294\ub370\uc11c \uc0bd\uc9c8\uc744 \ub9ce\uc774 \ud588\ub2e4. macvlan<\/code> \uacfc \uac19\uc740 \ub124\ud2b8\uc6cc\ud06c\ub97c \uc774\uc6a9\ud558\uba74 docker service composition \uc790\uccb4\ub294 \uc26c\uc6cc\uc9c0\uaca0\uc9c0\ub9cc, \ubc30\uc815 \ubc1b\uc544\uc57c\ud558\ub294 real ip \uac00 \ub298\uc5b4\ub098\ub294 \ub4f1 \uc778\ud504\ub77c \uc124\uc815\uc774 \ucd94\uac00\ub3fc \uad00\ub9ac \ube44\uc6a9\uc774 \ucee4\uc9c8 \uac83 \uac19\uc544 \uae30\ubcf8 \ub124\ud2b8\uc6cc\ud06c\uc5d0 \ud3ec\ud2b8 \ub9f5\ud551\uc73c\ub85c\ub9cc \ud574\uacb0\ud558\uace0\uc790 \ud588\ub2e4. Simple is Best.<\/del><\/p>\n
    \uae30\ubcf8 bridge<\/code> \ub124\ud2b8\uc6cc\ud06c\uc5d0\uc11c, \ub3c4\ucee4 \ucee8\ud14c\uc774\ub108\ub4e4\uc740 \uac00\uc0c1 \ub124\ud2b8\uc6cc\ud06c \uc778\ud130\ud398\uc774\uc2a4\ub97c \uc0dd\uc131\ud574 \uadf8 \uc548\uc5d0\uc11c \ub180\uae30\ub54c\ubb38\uc5d0 \ud638\uc2a4\ud2b8 \uc678\ubd80\uc5d0 \uc11c\ube44\uc2a4\ub97c \uc5f4\uc5b4\uc8fc\uae30 \uc704\ud574\uc11c\ub294 NAT\ub97c \uc774\uc6a9\ud574 \ud3ec\ud2b8\ud3ec\uc6cc\ub529\uc744 \ud574\uc918\uc57c\ud558\ub294\ub370, \uac01 \ucee8\ud14c\uc774\ub108 \uc11c\ube44\uc2a4\ub4e4\uc774 \uc790\uae30 \uc790\uc2e0\uc744 ip \uc8fc\uc18c\ub85c\ub9cc \uc54c\ub824\uc8fc\uba74 \ud3ec\ud2b8\ud3ec\uc6cc\ub529\uc744 \ud574\uc8fc\ub354\ub77c\ub3c4 \uc18c\uc6a9\uc774 \uc5c6\ub2e4.<\/p>\n
    \ub2e4\ud589\ud788 redis 6.2 \ubc84\uc804\ubd80\ud130 hostname\uc744 ANNOUNCE_IP<\/code> \ub97c \uc124\uc815\ud560 \uc218 \uc788\ub3c4\ub85d \uac1c\uc120\ub3fc docker \uc704\uc5d0\uc11c \uac01 \ub178\ub4dc\ub4e4\uc774 \uc11c\ub85c\ub97c \uc2dd\ubcc4\ud560 \uc218 \uc788\uc5c8\uace0 \ub2e4\uc74c\uacfc \uac19\uc774 \ud658\uacbd\ubcc0\uc218\ub97c \uc0ac\uc804 \uc124\uc815\ud574\uc11c PoC \uc6a9 redis \ub97c \uad6c\uc131\ud560 \uc218 \uc788\uc5c8\ub2e4.<\/p>\n
      redis-sentinel:\n    image: 'bitnami\/redis-sentinel:latest'\n    environment:\n      - ALLOW_EMPTY_PASSWORD=yes\n      - REDIS_MASTER_HOST=airflow-3\n      - REDIS_MASTER_PORT_NUMBER=6379\n      - REDIS_MASTER_SET=mymaster\n      - REDIS_SENTINEL_QUORUM=2\n      - REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=5000\n      - REDIS_SENTINEL_ANNOUNCE_HOSTNAMES=yes\n      - REDIS_SENTINEL_ANNOUNCE_IP=airflow-1\n    ports:\n      - 26379:26379<\/code><\/pre>\n
    \uadf8\ub9ac\uace0 \uac01 taskinstance \uc758 \uc791\uc5c5 \ub85c\uadf8\ub294 worker node \uc5d0\ub9cc \uc874\uc7ac\ud558\uae30\ub54c\ubb38\uc5d0 webserver \uc5d0\uc11c \ud574\ub2f9 \ub85c\uadf8\ub97c \ud655\uc778\ud558\ub824\uba74 celery worker \uc5d0\uc11c 8793 \ud3ec\ud2b8\ub85c \uc11c\ube44\uc2a4\ud558\ub294 \ubbf8\ub2c8 \uc6f9\uc11c\ubc84\uc5d0 \uc811\uadfc\ud560 \uc218 \uc788\uc5b4\uc57c\ud55c\ub2e4.<\/p>\n
      airflow-worker:\n    <<: *airflow-common\n    command: celery worker\n    hostname: worker-1\n    ports:\n      - 8793:8793<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
    airflow 2.0 \uc5d0\uc11c \uacf5\uc2dd\uc801(?)\uc73c\ub85c HA \ub97c \uc9c0\uc6d0\ud55c\ub2e4. 1.x \ubc84\uc804\uc5d0\uc11c\ub294 scheduler \ud504\ub85c\uc138\uc2a4\ub97c \ub3d9\uc2dc\uc5d0 \uc5ec\ub7ec\uac1c \uad6c\ub3d9\ud558\uba74 \uc791\uc5c5\uc774 \uc911\ubcf5 \uc2e4\ud589\ub420 \uc218 \uc788\ub294 \uc704\ud5d8\uc774 \uc788\uc5c8\uc11c \ubb34\uc870\uac74 single \ub85c \uc6b4\uc601\ud574\uc57c\ud588\uace0 \uc774 \ub54c\ubb38\uc5d0 \uac00\uc6a9\uc131\uc744 \ubcf4\uc7a5\ud560 \uc218 \uc5c6\uc5c8\ub294\ub370 2.x \uc5d0\uc11c\ubd80\ud130\ub294 row-level locking \uc744 \uc774\uc6a9\ud574\uc11c multiple scheduler \uc774\uc6a9\uc774 \uac00\ub2a5\ud558\ub3c4\ub85d \uac1c\uc120\uc774 \ub410\ub2e4. (\ub300\uc2e0 SKIP LOCKED \ub610\ub294 NOWAIT \uad6c\ubb38\uc744 \uc9c0\uc6d0\ud558\ub294 mysql8.0 \uc774\uc0c1\uc744 \uc368\uc57c\ud558\ub294 \uc81c\uc57d\uc870\uac74\uc774 \uc788\ub2e4.) […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[199,162],"tags":[],"class_list":["post-1724","post","type-post","status-publish","format-standard","hentry","category-airflow","category-docker"],"_links":{"self":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/1724","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/comments?post=1724"}],"version-history":[{"count":0,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/1724\/revisions"}],"wp:attachment":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/media?parent=1724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/categories?post=1724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/tags?post=1724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}